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To  the  President  of  the  Senate  and  the 
Speaker  of  the  House  of  Representatives 

Contracting  for  computer  software  development  can  be 
an  effective  alternative  to  software  development  by  Federal 
employees.  However,  the  effectiveness  of  such  contracting 
depends  on  very  careful  contract  management. 

This  report  discusses  the  problems  that  Federal  agencies 
have  encountered  in  contracting  for  computer  software  devel¬ 
opment  and  recommends  means  of  improving  such  contracting. 

We  are  sending  copies  of  this  report  to  the  Secretary 
of  Commerce  and  to  the  Administratorof  General  Services. 

Comptroller  General 
of  the  United  States 


COMPTROLLER  GENERAL'S  CONTRACTING  FOR  COMPUTER 

REPORT  TO  THE  CONGRESS  SOFTWARE  DEVELOPMENT~MORE 

MANAGEMENT  ATTENTION  COULD 
AVOID  WASTING  ADDITIONAL 
MILLIONS 


DIGEST 

Many  Federal  agencies  have  computer 
programs-“called  software  in  the  data  proc~ 
essing  industry — developed  by  outside 
sources.  These  sources  may  be  either  pri¬ 
vate  firms  or  other  Federal  agencies. 

Although  new  software  is  often  developed 
successfully  by  outside  sources,  GAO  found 
that  too  many  contracts  experience  large 
cost  overruns  and  lengthy  delays,  and  agen¬ 
cies  may  be  dissatisfied  with  the  product. 

Agencies  contract  with  outside  sources  for 
custom  software  development  for  various 
reasons.  For  instance,  they  do  not  have 
enough  staff  or  expertise  to  develop  it 
themselves,  or  they  can  get  it  at  lower 
cost. 

GAO  sent  questionnaires  to  163  software 
contracting  firms  and  113  Federal  project 
officers  who  had  experience  with  software 
development  contracts  to  identify  what  had 
caused  trouble  and  what  might  be  done  to  im- 
improve  development  efforts.  Certain  things 
causing  problems  for  both  contractors  and 
agencies  were  common  to  all  reviewed  con¬ 
tracts  that  had  trouble. 

GAO  examined  nine  cases  of  software  devel¬ 
opment  in  detail.  Eight  had  problems,  but 
their  overall  performance  cannot  be  taken 
as  representative — some  came  to  GAO's  atten¬ 
tion  because  they  were  failures.  Neverthe¬ 
less,  the  cases  illustrated  many  of  the  same 
causes  of  difficulty  that  GAO's  questionnaire 
respondents  had  identified. 

Only  one  of  the  nine  cases  yielded  software 
that  could  be  used  as  delivered.  The  com¬ 
bined  total  costs  and  development  times  of 


FGMSD-80-4 

Tear  Sheet.  Upon  removal,  the  report 
cover  date  should  be  noted  hereon. 

i  (M 

the  nine  cases  increased  from  estimates  of 
$3.7  million  and  10.8  years  to  an  actual 
cost  of  $6.7  million  and  an  actual  duration 
of  20.5  years. 

COMMON  CAUSES  OF  SOFTWARE 
development  contracting  problems 

Federal  agencies  contract  for  software  de¬ 
velopment  with  little  specific  guidance. 

This  circumstance  was  common  to  almost  all 
revi^ewed  contracts.  (See  p*  15.) 

Agencies  also  overestimate  the  stage  of  sys¬ 
tem  development  they  have  reached  before 
they  contract.  They  overestimate  the  com¬ 
pleteness  of  their  own  work,  such  as  ana¬ 
lyzing  user  requirements,  before  they  con¬ 
tract  for  software  development.  Often,  an 
agency's  preliniinairy  work  is  inadequate  and 
must  be  done  again  by  the  contractor. 

Overestimating  its  own  preliminary  work  can 
jin  agency  into  issuing  inappropriate 
contracts  and  using  inadequate  criteria  for 
contractor  performance.  (See  p.  17.)  By 
failing  to  stipulate  what  constitutes  sat¬ 
isfactory  performance  by  the  contractor, 
agencies  reduce  the  likelihood  that  the  de¬ 
livered  software  will  be  satisfactory.  The 
lack  of  a  good  contractual  description  of 
what  the  contractor  is  to  do  makes  it  diffi¬ 
cult  for  the  agency  to  claim  poor  contractor 
performance.  (See  P«  20.) 

Agencies  quickly  overcommit  themselves  and 
fail  to  control  contractors  through  strict 
phasing.  They  will  sometimes  commit  them¬ 
selves  to  the  entire  software  development, 
including  writing,  testing,  and  delivering 
the  computer  programs  before  they  even  have 
the  user  requirements — what  the  software  is 
to  do — clearly  identified.  In  such  situa¬ 
tions,  a  phased  contract,  initially  commit¬ 
ting  the  agency  only  to  an  analysis  and 
sign  phase,  and  then  proceeding  only  if  the 
first  phase  proves  satisfactory,  would  be 
much  more  suitable.  (See  p.  21.) 

Agencies  do  not  manage  software  development 
contracts  during  execution.  Management  fail¬ 
ures  while  the  work  is  being  done  included 
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excessive  changes  and  afterthoughts,  failure 
to  inspect  intermediate  stages  of  the  work, 
and  failure  to  require  progress  reports  from 
the  contractor.  In  one  case,  officials 
stated  that  they  could  not  review  the  work 
because  it  was  being  done  on  the  contractor's 
premises.  (See  p.  23.) 

In  contracts  that  have  problems,  contractual 
testing  requirements  are  often  sketchy  or  ab¬ 
sent.  Agencies  accept  and  pay  for  software 
without  adequately  inspecting  and  testing  it. 
Contractors  identified  inadequate  agency  test 
data  as  a  frequent  source  of  problems.  Fail¬ 
ures  to  inspect  test  output  and  documentation 
products  also  occur.  In  one  case,  the  con¬ 
tract  called  for  the  use  of  one  programming 
language  but  the  delivered  programs  were  writ¬ 
ten  in  another.  The  contractor  still  got  paid. 
(See  p.  24. ) 

Some  problems  occur  because  agencies  fail  to 
establish  a  single  focal  point  for  communi¬ 
cation  with  contractors.  Communications 
difficulties  and  delays  occur  when  contrac¬ 
tors  have  ho  identified  single  source  for 
answers  or  proposed  changes  and  interpreta¬ 
tions  of  requirements.  (See  p.  25.) 

GAO  also  found  that  problems  arise  because 
agencies  do  not  adequately  specify  or  en¬ 
force  contract  clauses  for  recovery  in  the 
event  of  poor  performance  by  the  contractor, 
and  contractors  frequently  fail  to  provide 
adequate  software  documentation. 

RECOMMENDATIONS 

GAO  recommends  that  the  Secretary  of 
Commerce,  through  the  National  Bureau  of 
Standards,  and  the  Administrator  of  General 
Services  issue  specific  guidelines  to  assist 
Federal  agencies  in  recognizing  and  dealing 
with  the  unique  factors  added  to  custom 
software  development  when  it  is  done  by 
contract.  The  following  areas  should  be 
covered ; 

-^Internal  agency  management  practices 
necessary  to  write,  manage,  and  monitor 
software  development  contracts. 
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— Specific  instructions  on  how  to  tailor 
software  development  contracts  to  the 
stai^e  of  system  development  that  an 
agency  is  in  at  the  time  it  lets  a  con¬ 
tract. 

— Guidance  on  contract  stipulations  regarding 
the  phasing  of  the  software  development. 

— Guidance  on  the  review  and  approval  pro¬ 
cedures  agencies  should  follow  at  the  end 
of  each  phase  of  software  development. 

— Guidance  on  performance  specifications  to 
be  included  in  the  contract  to  clarify 
quality  requirements  for  the  software. 

“The  importance  of  requiring  the  software 
contractor  to  have  a  formal  quality  assur¬ 
ance  program  that  is  documented  and  sub¬ 
ject  to  audit. 

— The  degree  of  definition  required  to  prop¬ 
erly  define  such  things  as  (1)  documenta¬ 
tion  standards,  (2)  adherence  to  program¬ 
ming  language  standards,  (3)  acceptance 
testing  procedures,  and  (4)  satisfactory 
performance  by  the  contractor. 

— ^^How  to  handle  changes  in  the  software 
being  developed  with  minimal  disruption. 

— How  to  ensure  that  the  contractor  follows 
sound  system  development  practices. 

— The  effective  use  of  contract  clauses 
which  would  deny  payment  in  case  of  poor 
performance  by  the  contractor. 

The  above  recommendation  could  be  achieved 
to  a  large  extent  if  the  National  Bureau  of 
Standards  and  General  Services  Administra¬ 
tion  designed  a  series  of  model  contracts 
containing  detailed  clauses  on  such  items 
as  documentation,  phasing,  and  testing.  A 
full  explanation  of  their  need  and  value 
should  accompany  these  clauses.  Agencies 
could  extract  relevant  clauses  and  construct 
contracts  to  fit  their  particular  situations. 
Such  model  contracts  are  recommended  primar¬ 
ily  as  aids  to  agency  software  development 
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contract  management,  but  they  would  also  aid 
the  agency  procurement  function  in  general. 

GAO  also  recommends  that  Federal  agencies 
involved  in  software  development  contracting 
train  project  managers  in  such  overall  skills 
necessary  to  manage  those  contracts  as  soft¬ 
ware,  contracting,  and  management.  Agencies 
should  also  take  appropriate  action  in  each 
phase  of  software  development  contracting. 

GAO  offers  a  provisional  checklist  for  con¬ 
tracting  for  software  development  to  outline 
appropriate  action  to  be  taken  for  each  phase 
(See  app.  I.) 

In  written  comments  on  the  report,  the  Gen¬ 
eral  Services  Administration  and  the  Depart¬ 
ment  of  Commerce  generally  agreed  with  the 
conclusions  and  recommendations.  (See  p.  31.) 
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CHAPTER  1 


INTRODUCTION 


FEDERAL  USE  OF  COMPUTERS 
AND  SOFTWARE  IS  EXTENSIVE 

The  Federal  Government  is  the  world's  largest  user  of 
automatic  data  processing  (ADP)  resources,  incurring  costs 
that  have  been  estimated  at  over  $10  billion  per  year  and 
which  continue  to  increase.  The  General  Services  Administra¬ 
tion's  (GSA's)  ADP  summary  for  fiscal  1978  reported  that  the 
Government  owns  or  leases  over  12,100  computers.  These  com¬ 
puters  are  used  to  process  a  variety  of  applications  ranging 
from  delivering  health  and  welfare  services,  to  administering 
social  security  and  veterans'  benefits,  to  exploring  space, 
to  analyzing  and  reporting  on  the  military. 

Computer  programs — generally  referred  to  in  the  industry 
as  software— are  what  make  all  these  computers  run.  A  com¬ 
puter  without  programs  is  like  a  phonograph  without  records— 
it  won't  play. 

Recent  industrywide  estimates  predicted  that  organiza¬ 
tions,  rather  than  have  their  employees  write  computer  pro¬ 
grams,  would  increase  their  spending  for  readymade  programs. 
For  example,  indications  are  that  in  1978,  pu^l^c  and  private 
organizations  may  have  spent  about  $2.6  billion  on  programs 
developed  by  outside  sources.  If  realized,  this  amount  will 
be  a  27-percent  increase  over  1977.  We  estimate  that  about 
800  independent  software  suppliers  operate  in  the  United^ 
States,  and  experts  have  predicted  a  five-fold  increase  in 
jobs  in  the  software  industry  by  1985.  Some  of  the  software 
suppliers  sell  ready-made  software;  others  contract  to  develop 
custom-built  software  for  clients.  1/  We  have  numerous  indi¬ 
cations  that  having  software  developed  by  outside  sources  and 
using  ready-made  software  are  accepted  and  successful  prac¬ 
tices  in  the  private  sector. 

Computer  software  development  has  historically  been  a 
problem  and  is  further  complicated  by  contracting  for  it. 
Literature  on  the  subject  contains  many  discussions  of  soft¬ 
ware  development  projects  that  were  late,  cost  too  much,  or 


1/The  difference  between  readymade  and  custom-developed 
“  software  is  analogous  to  the  difference  between  a  ready- 
to-wear  suit  and  a  made-to-order  suit. 
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failed  completely.  Causes  cited  for  these  problems  have  in¬ 
cluded  the  difficulty  of  defining  the  work  to  be  done,  changes 
to  the  scope  of  the  work  during  the  project,  and  lack  of  coromu 
nication  between  ADP  users  and  computer  specialists. 

FEDERAL  AGENCIES  SPEND  A  LOT  ON 
CONTRACTING  FOR  SOFTWARE  DEVELOPMENT 


Although  few  reliable  statistics  are  available,  it  is 
estimated  that  Federal  agencies  contract  for  several  hundred 
million  dollars  of  computer  software  development  by  software 
vendors  and  by  other  Government  agencies  annually.  For 
example,  the  GSA  ADP  summary  reported  that  about  $254  mil¬ 
lion  was  spent  in  fiscal  1978  for  contract  systems  analysis, 
design,  and  programming  in  the  general  management  cateqorv 
alone.  1/  ^  ■ 

Federal  agencies  usually  contract  for  software  develop¬ 
ment  because  they  lack  the  staff  or  skill  to  develop  it  in- 
house.  The  goal  of  such  contracting  is  to  promptly  deliver 
computer  programs  which  (1)  automate  necessary  tasks  for  the 
agency,  (2)  are  usable  as  delivered,  (3)  have  reasonable 
operating  and  maintenance  costs,  and  (4)  are  written  so  they 
can  be  easily  modified  later  to  meet  changing  requirements. 
Reasonable  operating  and  maintenance  costs  require  that  the 
programs  be  skillfully  written  (to  minimize  their  costs), 
thoroughly  tested  for  correctness,  and  well  documented  for 
ease  of  operation  and  interpretation.  If  later  modification 
is  to  be  made  with  ease,  the  programs  must  be  well  documen¬ 
ted  for  the  maintenance  programmers  who  will  modify  them. 

We  undertook  this  study  to  determine  the  extent  of 
problems  in  Federal  software  contracting,  to  identify  some 
of  the  common  underlying  causes  of  problems  in  this  area 
and,  if  possible,  to  recommend  means  of  improvement. 

ROLES  OF  VARIOUS  AGENCIES 


The  basic  law  governing  Federal  ADP  management  is  the 
Brooks  Act,  Public  Law  89-306.  Under  this  act,  the  General 
Services  Administration  is  responsible  for  procuring  and 
maintaining  Federal  ADP  resources.  GSA  receives  technical 
advice  from  the  Secretary  of  Commerce,  primarily  through  the 


^/This  figure  includes  contract  services  for  which  there  was 
no  deliverable  product  as  well  as  for  contracts  with  a  pro- 
duct,  but  does  not  include  software  developed  for  special- 
purpose  computers,  such  as  those  included  in  weapons 
systems. 
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National  Bureau  of  Standards  (NBS),  and  both  of 

agencies  receive  fiscal  and  policy  guidance  from  the  Office 

of  Management  and  Budget  (0MB). 


In  our  role  of  aiding  the  Congress,  we  are  concerned 
with  the  management  of  Federal  ADP  and  with  computer  soft¬ 
ware  development  as  a  frequent  and  expensive  activity.  1/ 
Our  past  reports  to  the  Congress  have  recommended  improve¬ 
ments  in  ADP  management  both  governraentwide  and  at  specific 
agencies. 

SCOPE  OF  REVIEW 

In  this  review,  we  concentrated  on  contracts  for  the 
development  of  custom-built  computer  programs  for  business 
and  administrative  purposes.  The  exception  is  our  fifth 
case  in  which  the  contractor  developed  a  compiler,  y  This 
review  did  not  address  military  or  special  management  con¬ 
tracts,  such  as  those  in  which  the  computers  and  their  pro¬ 
grams  are  parts  of  a  weapons  system,  nor  did  it  address 
readymade  software  purchases. 


We  did  the  following  work  in  our  review. 


—we  administered  two  nationwide  questionnaires:  one  to 
163  software  contractors,  the  other  to  113  Federal 
data  processing  personnel  with  software  contracting 
experience.  The  questionnaires  asked  contractors  and 
agency  personnel  to  (1)  identify  those  problem  areas 
they  experienced  in  software  contracting#  (2)  identify 
the  causes  of  those  problems,  and  (3)  evaluate  pro¬ 
posed  solutions. 

identified  and  analyzed  nine  cases  where  software 
development  was  contracted  for  with  Federal  funds. 

Some  were  brought  to  our  attention  because  they  were 
problem  cases.  We  evaluated  them  and  attempted  to 
determine  the  causes  of  any  problems  identified. 
Narratives  on  the  cases  are  included  in  appendix  II. 


l/House  Committee  on  Government  Operations,  "Administration 
“  of  Public  Law  89-306,  Procurement  of  ADP  Resources  by 
the  Federal  Government,"  Oct.  1,  1976. 

2/A  compiler  is  a  computer  program  which  translates  state- 
”  ments  in  a  programming  language  (written  by  a  person)  into 
the  internal  code  of  the  computer.  The  results  of  such 
translation  is  what  actually  makes  the  computer  perform 
the  required  tasks. 
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— -We  contacted  NBS  and  GSA  to  determine  what  guidance 
they  had  issued  to  Federal  agencies  to  help  them  suc¬ 
cessfully  contract  for  software. 

— We  examined  agencies'  internal  guidelines  to  determine 
the  adequacy  of  procedures  to  be  followed  in  contract¬ 
ing  for  computer  software  development. 

— We  researched  recent  studies  and  publications. 

--We  developed  a  provisional  checklist  of  agency  man¬ 
agement  actions  to  improve  software  development  con¬ 
tracting.  We  feel  that  this  checklist  will  be  helpful 
to  those  preparing  to  contract  for  software  develop¬ 
ment.  (See,  ,app»  I ) . 


CHAPTER  2 

SOME  SOFTWARE  DEVELOPMENT  CONTRACTS  EXPERIENCE 

EXTRA  COSTS*  DELAYS#  AND  OTHER  PROBLEMS 

We  found  that  some  software  development  contracts  made 
by  Federal  agencies  have  experienced 

overruns,  user  dissatisfaction,  and,  sometimes,  ^5  ^22=  ^ 

resulted  in  software  that  never  worked  in  spite  , 

time  and  money  spent  on  it  by  agency  programmers  after  the 

contractor  had  left. 

in  this  chapter,  we  will  discuss  ^^e  characteristics  of 
custom  software  development,  why  contracting 
its  problems,  and  the  conditions  under  which  contracting  is 
done  that  we  found  in  our  investigation. 

SOFTWARE  DEVELOPMENT  TRADITIONALLY 
HAS  BEEN  A  PROBLEM  AREA 

Software  development 

The  complete  life  cycle  of  computer  software  is  divided 
into  requirements  analysis,  system  design  or 
development,  and  operation.  Development  time  ends  when  the 
sortia?fu'ln  promotion.  1/ 

development  projects  have  experienced  one  or  more  of  the 
following  problems: 

—They  have  cost  more  than  expected  and  have  run  late. 
Adding  more  people  to  late  projects  has  not  often 
restored  them  to  schedule. 

—The  first  production  versions  delivered  were 

prototypes  in  conventional  engineering  terms.  Beside 
the  programs  themselves  being  prototypes,  their  docu¬ 
mentation  was  often  sketchy  or  missing. 

—Because  prototypes  were  delivered,  the  operational 
costs  of  fixing  and  modifying  them  have  typically 
been  as  great  as  the  development  costs  and  sometimes 

far  greater. 


1/Software  that  is  in  production  is  doing  the  task  for  which 
“  it  was  written;  for  example,  a  payroll  program  is  in  pro¬ 
duction  when  it  is  printing  checks. 
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Several  factors  contributed  to  the  situation.  First, 
the  invisible  nature  of  both  the  work  process  and  its  product 
made  software  projects  very  difficult  to  manage  and  predict. 
Second,  the  explosive  growth  of  the  use  of  computers  created 
a  great  demand  for  new  programmers,  most  of  whom  were  self- 
taught  on  the  job;  and  frequently,  low  productivity  and  poor 
product  quality  resulted.  Third,  there  was  little  idea  then 
of  how  to  train  programmers  properly.  Fourth,  a  tradition 
grew  that  programmers  were  secretive  craftspersons  whose 
products,  during  development,  were  their  own  property. 

Software  operation 

The  operational  phase  lasts  from  the  beginning  of  pro¬ 
duction  to  the  time  the  software  is  replaced  or  discarded. 
During  this  time,  costs  beyond  normal  operating  costs  may  be 
incurred  to  (1)  correct  errors  in  the  software,  (2)  modify 
the  software  so  that  new  functions  can  be  added,  (3)  tune 
the  software  to  reduce  excessive  operating  costs,  or  (4)  con¬ 
vert  the  software  to  run  on  another  computer. 

Corrections  of  errors  and  modifications  are  commonly 
combined  under  the  term  "maintenance."  Estimates  of  pro¬ 
grammer  time  spent  maintaining  software  that  is  developed 
traditionally  have  ranged  in  numerous  organizations  from  20 
to  80  percent  of  total  programming  effort. 

Historically,  projects  both  to  develop  original  software 
and  to  convert  current  software  have  often  been  completed 
later  or  at  higher  cost  than  predicted,  or  both.  Many  causes 
contribute  to  this  situation — some  are  managerial,  some 
sociological,  and  some  technical. 

In  the  managerial  category,  the  ability  to  measure  and 
predict  software  projects  has  been  lacking.  Workers  who 
produce  software  are  commonly  considered  to  be  difficult  to 
retain  and  to  manage  effectively.  Customers  for  whom  the 
software  is  created  (end  users)  often  do  not  have  a  clear 
understanding  of  the  process  or  function  they  want  automated 
when  the  software  development  begins.  Changes  requested 
after  projects  have  started,  which  seem  trivial  to  the  cus¬ 
tomers,  have  often  required  major  rework  and  have  resulted 
in  delays  and  increased  costs. 

In  the  sociological  category,  our  previous  software 
conversion  review  1/  found  that  many  computer  programmers 


l/"Millions  in  Savings  Possible  in  Converting  Programs  from 
One  Computer  to  Another,"  FGMSD-77-34,  Sept.  15,  1977. 
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view  themselves  as  craftspersons,  with  strong  feelings  that 
"I'd  rather  do  it  myself."  Traditionally/  the  programmers 
secretive  attitude  was  indulged  by  management  because  pro- 
greunmers  were  scarce  and  difficult  to  retain. 

Technical  problems  result  from  the  need  to  meet  dead¬ 
lines— programs  are  often  designed  and  written  hastily ,_and 
are  tested  and  documented  inadequately  or  not  at  all.  Thus, 
quality  is  sacrificed  to  urgency.  Documentation--material 
prepared  to  explain  a  computer  program — is  often  deferred 
until  after  the  program  is  running  and  sometimes  is  never 
completed.  When  programs  are  later  modified  or  converted, 
the  work  is  usually  done  by  someone  other  than  the  originator 
If  documentation  is  missing,  incomplete,  or  ® 

deal  of  the  original  development  work  often  must  be  repeated 
by  the  person  modifying  or  converting  the  program  before  that 
pLson  can  hope  to  modify  it  or  convert  it  successfully. 


CONTRACTING  FOR  SOFTWARE 
DEVELOPMENT  ADDS  TO  ITS  DIFFICULTY 


The  development  of  original  software  which  meets  users 
needs  has  a  tradition  of  managerial  and  technical 
even  when  the  programmers  and  analysts  developing  it  work  _ 
for  the  same  organization  as  the  users  who  need  it.  Several 
sources  of  difficulty,  as  described  below,  i 

the  software  is  developed  by  outsiders,  as  it  is  when  Federa 
agencies  contract  for  software  development. 

—The  problem  definition  and/or  user  requirement  must 
be  defined  so  that  "outsiders"  can  understand  it. 


— Contracting  introduces  an  extra  communication  link 
between  the  software  users  and  developers. 

—The  capability  of  the  contractor  should  be  checked 
and  verified  before  the  contract  is  let. 


—Extensive  acceptance  testing  criteria  and  test 

should  be  developed  before  the  contract  is  released 
and  acceptance  tests  contractually  required.  Testing 
is  also  needed  with  software  developed  in-house,  but 
contracting  emphasizes  the  need  for  it. 

—Contractor  personnel  must  be  informed  about  agency 
operations. 

—Agency  management  must  control  the  quality  of  work 
done  outside  the  agency. 
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— First-hand  observation  of  progress  is  more  difficult 
for  the  agency. 

— The  contractor's  knowledge  of  the  software  is  lost 
when  the  finished  software  is  brought  in-house.  Even 
with  good  documentation,  which  is  sometimes  lacking, 
agency  personnel  must  become  familiar  with  the  pro¬ 
grams. 

— Acquisition  of  software  from  a  contractor  requires  an 
agency  to  identify  and  meet  all  applicable  Government 
procurement  regulations. 

TOO  MANY  SOFTWARE  DEVELOPMENT 
CONTRACTS  HAVE  PROBLEMS 


Our  sources  of  information  indicated  that  many  software 
development  contracts  produce  software  that  is  useful  to  the 
customer.  However,  a  substantial  number  do  not.  Also,  even 
some  of  the  delivered  software  that  can  be  used  must  be  re¬ 
worked  by  agency  staff  after  delivery  before  it  works  satis¬ 
factorily.  Those  who  responded  to  our  questionnaire  indica¬ 
ted  that  problems  often  occurred  with  development  contracts. 
Our  case  studies  confirmed  these  problems  and  Illustrated 
the  situations  in  which  they  occur. 

Questionnaires  reported  common  problems 

Figures  1  through  4  show  the  frequency  ly  that  the 
questionnaire  respondents  reported  for  several  conditions. 
Figure  1  shows  what  the  respondents  said  about  dollar  over¬ 
runs:  21  percent  said  that  their  occurrence  was  "very  com¬ 
mon,”  29  percent  said  "fairly  common,"  and  only  6  percent 
would  say  they  "never"  occurred.  Figure  2  shows  that  our 
respondents  reported  frequent  calendar  overruns:  30  percent 
said  they  were  "very  common,"  32  percent  said  "fairly  common," 
and  only  about  2  percent  would  say  they  "never"  happened. 
Figure  3  shows  that  our  respondents  reported  that  even  though 
software  is  finally  delivered,  it  must  often  be  reworked: 
about  9  percent  said  that  problem  was  "very  cOTunon,"  35  per¬ 
cent  said  "fairly  common,"  and  only  6  percent  said  it  "never" 
occurred.  Figure  4  indicates  the  responses  to  the  worst 
situation — that  is,  software  was  paid  for  and  not  used,  for 
which  only  20  percent  said  it  "never"  occurred. 


1/In  this  question,  we  defined  very  common  as  over  75  per¬ 
cent  of  the  time,  fairly  common  as  51  to  75  percent  of  the 
time,  and  not  very  common  as  25  to  50  percent  of  the  time. 
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Figure  1 


"Software  development  has  dollar  overrun. 


Response 

Very  common 
Fairly  common 
Not  very  common 
Very  rare 
Never  occurs 
Don ' t  know 

Total 


Respondents 


[umber 

Percentage 

24 

21.2 

33 

29.2 

29 

25.7 

11 

9.7 

7 

6.2 

_9 

8.0 

113 

100.0 

Figure  2 

"software  development  has  calendar  overrun. 


Response 


Number 


Respondents _ 

Percentage 


Very  common 
Fairly  common 
Not  very  common 
Very  rare 
Never  occurs 
Don ' t  know 


34 

36 

29 

9 

2 

3 


30.1 

31.9 

25.7 

8.0 

1.8 

2.7 


Total 


113  a/  100.0 


^Does  not  add  due  to  rounding 
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Figure  3 

"The  delivered  software  must  be  corrected  or  modified  by 
in-house  programmers  before  it  is  usable." 


Respondents 


Response 

Number 

Percentage 

Very  common 

10 

8.8 

Fairly  common 

39 

34.5 

Not  very  common 

40 

35.4 

Very  rare 

15 

13.3 

Never  occurs 

7 

6.2 

Don't  know 

_2 

1.8 

Total 

113 

100.0 

Figure  4 

"The  software  is  paid 

for  but  never 

used. " 

Response 

Number 

Respondents 

Percentage 

Very  common 

Fairly  common 

4 

3.6 

Not  very  common 

18 

16.1 

Very  rare 

64 

57.1 

Never  occurs 

23 

20.5 

Don't  know 

_3 

2.7 

Total 

112 

100.0 

Case  studies  confirmed  questionnaires 

We  examined  nine  cases  in  detail,  which  included  visits 
to  both  agency  and  contractor  sites,  examining  documents, 
and  interviewing  those  persons  involved  who  could  still  be 
reached.  We  must  caution  the  reader  that  several  cases  came 
to  our  attention  because  they  were  problems.  Therefore,  al¬ 
though  they  cannot  be  taken  as  representative  of  all  Federal 
software  development  contracts,  they  nevertheless  dramatically 
illustrate  software  development  contracting  problems. 


out  oases  also  provide  examples  of  the 

?lsT,  ?Lse  con?racts  averaged  delays 

of  about  75  percent  of  their  original  time 

nre  6  summarizes  conditions  that  were  common  to  the  case 

st!dier™rshows  how  they  relate  to  the  conditions  identi- 

fied  on  the  questionnaires. 


Figure  5 


NINE  SOFTWARE  DEVELOPMENT  CONTRACTS  TOTALING  $  6.8  MILLION: 

WHERE  THE  MONEY  WENT. 
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“ont'^acting  oases  which  had  probleias 
Identified  on  the  agency  questionnaire  i'rooAems 


Condition  identified 

Software  development 
has  dollar  overrun 

Software  development 
has  calendar  overrun 

The  delivered  software 
roust  be  corrected  or 
roodified  by  in-house 
prograroroers  before 
it  is  usable 

The  software  is  paid 
for  but  never  used 

Atteropts  to  re— work 
and/or  use  the  soft¬ 
ware  in-house  were 
tried  but  failed 

Software  roet  acceptance 
testing  as  delivered 


Case  Number 
4 _ 5  6 


that  Prob?a:ri°"slft”re"LvaL^l:t“o^t‘r«t!^ro""  "T 

ig;  .s  a;  a'.».  ilaSr 

SOFTWARE  DEVELOPMENT  CONTRACTING 

problems  have  undesirable  effects 

softiarl  paychecks, “irthr?easSi  whyV^ 
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automated  at  all#  or  may  continue  to  be  done  with 
older#  more  costly  software  that  was  supposed  to  be 
replaced. 

_ Additional  cost  before  delivery.  The  contractor  often 

gets  paid  more  than  predicted. 

—Additional  cost  fl^ht  after  delivery.  Even  when  a 
contractor  delivers  software#  in-house  programmers 
may  need  to  modify  it  before  it  is  satisfactory  fo'^^ 
production.  This  modification  adds  cost  and  may  fail. 

The  delayed  effects  which  ..can  pdciir  include: 

-i-years  of  maintenance’  and  lyiodif  ication  that  cost  more 
than  they  should  when  the;:  ^  programs  are 

pOoirly  organized  or  poorly  documented. 

_ Slowed  response  to  later  requests  for  modification. 

Later  requests  for  modification  to  what  the  software 
does  for  the  users  can  come  from  the  users  themselves 
or  can  be  legislated.  Those  requests  are  delayed  when 
programs  are  poorly  organized  or  poorly  documented. 

Figures  7  and  8  show  what  the  Federal  questionnaire  re¬ 
spondents  said  about  two  indicators  of  delayed  effects.  Fig¬ 
ure  7  shows  that  many  of  our  respondents  report  difficult—  ^ 
to-modify  software  as  a  frequent  occurrence.  Figure  8#  while 
giving  a  somewhat  more  optimistic  picture#  still  shows  a  sig¬ 
nificant  occurrence  of  difficult— to— understand  software. 


Fiqure  7 

"The  delivered  software  is 

difficult 

to  modify. " 

- 1.. 

Respondents 

Response 

Number 

Percentage 

Very  common  • 

'  Fairly  common 
■  Nd^t  .vety^  GoirariQn  -  v- 
V.ety:;.rare’' 

Nevfet- 

Don't  kno«?  -  -o,  ■  X'-'’ 

6 

42 

43 

13 

5 

_4 

5.3 

37.2 

38.1 

11.5 

4.4 

3.5 

Total 

100.0 

13 


Figure  8 

"The  contractor's  programming  practices  are  such  that  the 
software  is  easily  understood  by  agency  programmers." 


Response 

Respondents 

Number 

Percentage 

Very  common 

16 

14.2 

Fairly  common 

71 

62.8 

Not  very  common 

17 

15.0 

Very  rare 

7 

6.2 

Never  occurs 

— 

Don't  know 

_2 

• 

GO 

Total 


113 


100.0 


CHAPTER  3 


MAJOR  CAUSES  OF  PROBLEMS  IN 
CONTRACT  SOFTWARE  DEVELOPMENT 

Among  the  software  development  contracts  that  have 
problems,  we  found  several  common  causes  of  those  problems, 
even  though  each  case  has  unique  features  of  its  own.  The 
responses  to  our  questionnaire  and  our  review  of  the  case 
studies  brought  several  of  these  causes  to  the  surface. 

We  found  that  agencies  contract  for  most  software  devel¬ 
opment  because  (1)  they  lack  enough  staff,  or  staff  with  the 
right  skills,  to  do  it  in-house  or  (2)  because  they  need  the 
software  sooner  than  in-house  staff  could  develop  it.  We  have 
numerous  indications  from  this  and  other  studies  that  agencies 
would  generally  develop  software  in-house  if  the  resources 
were  available.  However,  a  cost  advantage  is  sometimes  used 
to  justify  contracting. 

AGENCIES  NOW  CONTRACT  FOR  SOFTWARE 
DEVELOPMENT  WITH  LITTLE  GUIDANCE 

We  found  that  agency  staff  connected  with  software 
development  contracts  typically  have  little  guidance,  either 
from  central  agencies  or  from  their  own  agency  headquarters. 

Central  agency  guidance 

The  basic  responsibilities  of  the  central  agencies  are 
described  in  Public  Law  89-306,  the  Brooks  Act.  The  Office 
of  Management  and  Budget  provides  fiscal  and  general  over¬ 
sight  of  ADP  activity.  0MB  has  delegated  responsibility  to 
GSA  for  attaining  cost  effectiveness  in  the  selection,  acqui¬ 
sition,  and  utilization  of  ADP  resources.  The  National  Bureau 
of  Standards  is  assigned  the  task  of  developing  technical 
standards  and  guidelines.  0MB  guidance — since  the  Brooks  Act 
was  passed — has  indicated  that  NBS  is  also  responsible  for 
investigating  the  conduct  of  system  studies,  including  (1)  mon¬ 
itoring  their  performance  and  implementation,  (2)  preparing 
proposals,  specifications,  and  system  requirements,  and  (3) 
continually  evaluating  installation  and  system  performance. 

We  asked  NBS  and  GSA  what  guidance  they  had  provided 
Federal  agencies  in  the  specific  management  aspects  of  soft¬ 
ware  contracting.  NBS  representatives  informed  us  that  while 
their  responsibilities  involve  management  and  contracting 
activity,  their  agency's  emphasis  has  been  and  will  continue 
to  be  on  the  technical  aspects,  such  as  the  standardization 
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of  the  common  business-oriented  language  (COBOL)  for  Government 
use.  They  provided  us  with  a  primer  for  project  management 
and  quality  control ^  which/  although  it  makes  general  -refer* 
ences  to  contracting  situations,  for  the  most  part  contains 
system  development  instructions. 

GSA's  guidelines  pertaining  to  the  management  of  ADP 
resources  have  been  issued  in  the  form  of  Federal  Property 
Management  Regulations,  subpart  101-32,  while  policies  and 
procedures  on  procurement  of  and  contracting  for  commercially 
®^®ii®ble  software  are  set  out  in  Federal  Procurement  Regu¬ 
lation  1-4.11.  Our  review  of  these  guidelines  showed  they 
deal  almost  entirely  with  procurement  of  commercially  avail¬ 
able  software  and  not  the  specific  management  of  contracting 
for  custom  software  development.  Like  NBS,  GSA  has  also 
issued  system  development  or  project  management  publications 
for  agency  use,  but  they  do  not  deal  specifically  with  soft¬ 
ware  development  contracting. 

Guidance  at  the  agency  level 

We  asked  the  agencies  involved  in  our  case  studies 
about  the  guidance — if  any — they  have  issued  on  the  manage¬ 
ment  aspects  of  contracting  for  software  development.  Some 
agencies  furnished  us  policy  manuals  and  directives  on  soft¬ 
ware  development.  The  manuals  and  directives  were  primarily 
instructions  on  how  to  develop  systems,  with  little  specific 
information  on  how  to  contract  for  system  development. 

Only  one  agency  reviewed  differed  from  this  general 
pattern.  This  agency  has  issued  a  series  of  software  acqui¬ 
sition  management  handbooks  which  are  used  internally  as 
guidelines  when  contracting  for  software.  They  cover  such 
functions  as  (1)  monitoring  and  reporting  of  software  devel¬ 
opment  status,  (2)  statement  of  work  preparation,  and  (3) 
software  quality,  cost  estimation,  and  measurement.  These 
guidebooks  are  a  step  in  the  right  direction  toward  more' 
specific  management  guidance  on  software  contracting.  We 

hhat  they  could  serve  as  a  basis  for  publications  for 
NBS  to  disseminate  governmentwide. 

SOFTWARE  DEVELOPMENT  CONTRACTS  IN 
TROUBLE  HAVE  COMMON  CAUSES 

Besides  the  lack  of  specific  guidance  discussed  above, 
our  case  studies  and  the  responses  to  our  questionnaire  iden¬ 
tified  several  factors  which  were  at  the  root  of  the  problems 
With  many  software  development  contracts.  The  presence  of 
those  factors  in  our  case  studies  is  summarized  in  figure  9, 
and  the  cases  are  detailed  in  appendix  II.  Often,  several 
factors  were  present  in  one  case. 
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Agencies  overestimate  the  stage  of  system 
development  they  have  reached  before_ 
the  contract 

By  "stage  of  system  development,"  we  mean  the  point  the 
agency  has  rlaohed  in  the  series  of  steps  followed  in  devel- 
nn'ing  a  software  svstem.  For  a  number  of  reasons,  it  is 
critical  that  the  agency  be  accurate  in  determining  the  steps 
uSiriompleted  before  it  begins  the  contracting  process. 

_ If  the  agency  completes  some  system  development 

work  before  contracting,  the  contractor  will  pre¬ 
sumably  begin  from  that  point.  This  exact 
Lst  be  determined  to  get  realistic  cost  and  time 
estimates.  If,  after  the  contract  underway,  the 
aaencv’s  previous  work  is  discovered  to  be  less  ad 
vanced  than  initially  thought,  and  the  scope  of  the 
contractor's  work  increases,  costs  may  increase 
enough  to  destroy  any  cost/benefit  that  may  have 
originally  justified  the  contract. 

—The  point  to  which  system  development  P^^^ressed 
may  affect  the  type  of  contract 

For  instance,  if  the  agency  has  completed  the  detail 
design  of  its  system,  a  firm-fixed  price  contract 
for  the  programming  work  may  be  possible.  On  the 
other  hand,  if  the  agency  has  not  defined  re¬ 
quirements  and  has  no  system  design 
might  need  to  let  a  phased,  cost-plus-fixed-fee 
contract  with  proper  audit  clauses  because  the  exa 
scope  of . work  is  not  known. 

—If  work  done  by  the  agency  before  the  contract  is 
let  is  found  to  be  inadequate  after  the 
begins  work,  the  contractor  often  needs  to  begin 
agliS  with  a  different  approach.  When  this  happens, 

tL  tendency  is  to  try  to  save  «=  tS  fit 

of  the  work  already  done  and  to  modify  it  to  fit 
the  new  approach.  This  approach  nearly  always  com- 
promJlL  the  new  system  and  makes  it  less  efficient, 
causing  higher  operating  costs. 

--If  the  original  work  scope  must  be 

call  for  skills  the  contractor  does  not  have.  For 
instance,  a  contractor  may  agree  to 
only  to  find  that  design  work  is  needed  before  pro¬ 
gramming  can  be  started.  The  contractor  may  not  be 
qualified  to  do  the  necessary  design  work. 
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Figure  9 

Causes  of  problems  that  were  present  in  our  cases 

_ Case  number _ 

Cause  1  2  3  4  5  6  7  8 - 9 


Agency  overestimated  its  x  x 
own  state  of  progress 
when  it  let  the 
contract 

Incorrect  agency  manage¬ 
ment  action,  such  as 
using  inappropriate 
contract 

Agency  failed  to  specify  x 
requirements  adequately 

Agency  overcommitted 
itself 

Agency  failed  to  manage  x 
during  execution, 
including  excessive 
changes 

Agency  failed  to  x 

adequately  inspect 
and  test 


X  x 

X  X  X  X 

X  X  X  X  X 

X  X 

XX  X 

XX  XX 


In  several  of  the  case  studies,  the  agencies  accom¬ 
plished  considerably  less  than  they  thought  they  had  when 
they  let  the  contracts*  The  most  obvious  effects  were  delays 
and  extra  costs  because  the  contractor  had  to  perform  unex— 
pected  extra  work.  Other  effects  were  noted  which  substan¬ 
tiated  the  importance  of  an  agency  isolating  precisely  how 
far  it  had  progressed  in  its  system  development  before  con¬ 
tracting.  For  instance: 

Case  1,  the  system  concept  the  agency  originally 
proposed  was  changed  long  after  the  contract  com¬ 
menced.  An  analysis  of  the  product  delivered  under 
this  contract  showed  that  so  much  of  the  old  con¬ 
cept  remained  in  the  new  system  that  it  was  com¬ 
plicated  and  inefficient. 
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— In  Case  2,  a  contract  was  let  to  a  contractor 
because  he  was  experienced  in  developing  a  similar 
system  for  the  agency,  and  some  of  the  programs  in 
the  earlier  system  were  to  be  adapted  to  the  new 
one.  However,  the  agency  did  not  establish  the 
technical  feasibility  of  adapting  the  old  programs 
before  letting  the  contract,  and  when  they  found 
that  the  old  system  could  not  be  adapted  and  a  new 
design  concept  was  needed,  the  experience  for  which 
the  contractor  was  hired  was  useless  to  the  new 
approach.  As  a  result,  the  agency's  system  devel¬ 
opment  status  reverted  to  an  earlier  stage,  and  the 
scope  of  the  contract  became  far  greater  than  ori¬ 
ginally  intended. 

These  and  other  cases  provide  examples  pf  agencies'  failure 
to  determine  their  correct system  development  status.  (See 
app .XI.) 

Agencies  take  incorrect 

management  action 

Incorrect  actions  by  agency  management  can  condemn  a 
software  development  contract  before  it  is  even  started.  As 
discussed  above,  sometimes  the  incorrect  actions  are  based 
on  the  agency's  overoptimistic  assessment  of  the  work  it 
has  done. 

This  was  the  case  in  our  third,  fourth,  sixth,  and 
eighth  studies.  In  the  third  case  study,  the  agency  awarded 
a  fixed-price  contract  with  phased  development  but  did  not 
require  that  each  phase  be  approved  before  work  was  begun  on 
the  next.  This  type  of  contract  was  awarded  even  before  the 
the  user  requirements  had  been  identified — a  situation  for 
which  a  fixed-price  contract  is  not  appropriate.  In  the 
fourth  case,  the  agency  let  the  contractor  define  the  cri¬ 
teria  by  which  his  own  work  would  be  judged.  Agency  manage¬ 
ment  also  failed  to  derive  an  agency  consensus  on  system 
requirements.  This  failure  allowed  an  excessive  number  of 
reports  to  be  required — 188 — as  compared  to  much  smaller 
numbers  of  reports  required  in  several  other  accounting  sys¬ 
tems  for  which  44  was  the  largest  number. 

In  our  sixth  case,  the  agency  management  relied  upon 
discussions  with  the  contractor  to  define  requirements  and 
stated  in  the  contract  that  written  progress  reports  would 
not  be  necessary.  Since  no  agreement  about  the  requirements 
was  put  in  writing,  the  agency  later  had  no  way  to  require 
the  contractor  to  perform.  In  our  eighth  case,  the  agency 
management  failed  to  include  standards,  and  criteria  were 
written  without  adequate  preliminary  systems  analysis  work. 
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Failure  to  specify  what 

constitutes  satisfactory  performance 

It  is  important  to  specify  not  only  what  the  software 
is  to  do  but  how  well  it  is  to  do  it  and  how  well  it  is  to 
be  described.  Whether  the  specifics  of  the  software  are 
known  or  not,  some  general  requirements  and  constraints  can 
usually  be  identified  at  the  start.  Some  areas  where  we 
believe  that  performance  can  be  specified  are: 

1.  Growth  potential — software  may  need  to  be  modified 
to  handle  future  increases  in  workload. 

2.  Documentation  standards — specific  types  and  quality 
of  documentation  should  be  required. 

3.  Test  and  acceptance  criteria— how  software  must  per¬ 
form  before  it  will  be  accepted  must  be  identified. 

4.  Maximum  computer  resources  allowable — programs 
should  be  required  to  run  in  less  than  specified 
maximum  computer  time  and  capacity. 

5.  Maintenance — software  should  be  designed  and  written 
so  that  it  can  be  corrected,  changed,  or  modified 

as  simply  as  possible. 

6.  Transfer — software  should  be  designed  and  written 
to  facilitate  its  transfer  from  one  computer  to 
another. 

We  found  that  much  of  the  rework,  and  many  disputes 
over  whether  the  contractor  had  performed  according  to  the 
contract,  could  have  been  avoided  had  these  factors  been 
defined  at  the  outset.  In  our  first  case,  the  user  require¬ 
ments  that  the  agency  thought  were  adequate  were  actually 
useless.  Upon  discovering  this,  the  agency  allowed  the  con¬ 
tractor  to  develop  specifications.  The  lack  of  adequate  user 
requirements  forced  the  agency  to  make  many  changes  and  create 
delays  as  the  contract  progressed.  The  absence  of  testing 
left  the  agency  with  no  way  to  inspect  the  product. 

In  our  fourth  case,  the  agency  allowed  the  contractor  to 
develop  the  criteria  by  which  he  would  be  judged;  for  docu¬ 
mentation,  the  agency  merely  referred  the  contractor  to  agency 
standards.  In  our  ninth  case,  the  contract  required  that  doc¬ 
umentation  conform  to  county  standards,  which  were  nonexistent 
when  the  contract  was  signed.  The  contract  also  let  the  con¬ 
tractor  choose  between  two  programming  languages. 
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Agencies  over commit  themselves  too 

soon,  and  fail  to  control  contractors 

through  strict  phasing 

In  this  context,  phasing  means  stipulating  in  the  con¬ 
tract  that  work  be  done  in  logical  phases,  with  mandatory 
agency  approval  of  each  phase  before  proceeding  to  the  next 
phase.  When  properly  used,  phasing  provides  one  of  the 
strongest  methods  of  control  available  to  an  agency  that  is 
contracting  out  for  software  developmemt.  Phasing  provides 
the  agency  with  the  following  controls  and  advantages: 

— Initially,  phasing  helps  identify  critical  milestones 
which  can  be  used  to  construct  a  milestone  chart  and 
an  overall  timetable  for  the  development  project. 

Such  a  timetable  allows  the  agency  to  monitor  the  con¬ 
tractor's  progress  throughout  the  contract  period. 

—Phasing  allows  the  agency  to  assure  itself  that  the 
software  will  be  developed  in  a  systematic  and 
orderly  manner. 

— Phases  can  be  spaced  so  that  the  quality  and  accept¬ 
ability  of  the  contractor's  work  is  determined  before 
additional  funds  are  spent. 

Phasing  allows  the  agency  to  determine  at  the  end  of 
each  phase  that 

— the  contractor  is  following  sound  development 
practices; 

—the  contractor's  work  demonstrates  a  clear  under¬ 
standing  of  agency  requirements; 

— the  contractor's  proposals  are  technically  feasible; 
and 

— the  phase  under  review,  and  other  phases  completed 
to  date,  represent  an  adequate  base  to  support  the 
later  phases  of  development. 

The  agencies  that  were  the  subject  of  our  seventh  and 
ninth  case  studies  did  not  adequately  review  their  contrac¬ 
tors'  work.  In  our  seventh  case,  the  agency  committed  itself 
to  software  development  without  a  system  design,  i.e.,  a 
description  of  what  the  system  was  to  do.  A  CPA  firm  which 
later  reviewed  the  attempt  to  develop  the  software  said  that 
"the  system  was  never  really  designed."  Since  there  was  no 
design,  it  was  very  difficult  later  for  the  agency  to  withhold 
payment  for  poor  performance. 


21 


In  our  ninth  case,  the  county  committed  itself  to  a 
firm-fixed-price  contract  before  it-had  any  clear  idea  of 
what  its  system  was  to  do — that  is^^without  functional  speci¬ 
fications.  The  extreme  degree  of  commitment  of  this  type 
of  contract  was  entirely  inappropriate  in  this  situation. 

It  later  proved  impossible  to  require  the  contractor  to  fin¬ 
ish  the  work  for  the  fixed  price.  County  programmers  later 
spent  about  $17,000  trying  to  make  the  system  work  and  even¬ 
tually  the  county  was  able  to  use  the  system. 

The  completed  and  approved  phases  must  be  left  as  nearly 
intact  as  possible  so  that  the  phasing  concept  is  not 
destroyed.  (The  effects  of  excessive  changes  are  discussed 
further  in  a  following  section. )  While  the  case  studies  in¬ 
dicated  that  some  agencies  made  at  least  an  effort  to  phase 
the  contracts,  the  phasing  was  no.t'^Slways  satisfactorily 
accomplished.  We  saw  such  examples  of  nonphasing  practices 
as  work  being  done  in  the  programming  phase  before  the  design 
phase  was  approved  and  user  requirements  not  fully  determined 
even  though  work  had  begun  on  advanced  phases.  These  examples 
illustrated  the  agencies'  failure  to  use  phasing  as  a  manage¬ 
ment  technique  to  monitor,  control,  and  direct  contract  soft¬ 
ware  development.  S 

Agencies  do  not  manage  software 
development  contracts  during 
execution 

The  case  studies  showed  an  excessive  number  of  system 
changes  requested  by  the  agency;  the  changes  ranged  from 
adding  minor  requirements  to  changing  the  entire  system  con¬ 
cept.  In  some  cases  the  contractor  told  the  agency  that  mak¬ 
ing  the  changes  was  troublesome  or  unnecessary,  and  in  many 
cases  the  changes  were  made  well  into  the  contract  period. 
Also,  contractors  indicated  that  agency-initiated  changes 
in  work  scope  contributed  to  the  contract's  cost  and  time 
overruns.  In  addition,  changes  undermine  the  development 
effort  for  the  following  reasons: 

--Changes  are  not  usually  as  thoroughly  researched  as 
original  design  concepts  and  sometimes  have  unfore- 
,  seen  effects  on  other  parts  of  the  system. 

—As  mentioned  above,  effective  use  of  contract  phas¬ 
ing  can  be  destroyed  by  constantly  making  changes 
to  work  tha,t:was  completed  and  approved  in  earlier 

;■ 'phases.'  .. 

—— Under ^ cond itions  of, constant  change  the  agency  will 
find  it  difficult  to  determine  the  exact  status  of 
the  project  at  any  given  time.  Consequently,  it 


may  become  aware  of  potential  cost  and  time  over¬ 
runs  much  later  than  it  would  otherwise. 

— Excessive  changes  make  it  difficult  to  hold  the 
contractor  responsible  to  perform  according  to  the 
initial  terms  of  the  contract. 

A  thorough  review  of  each  phase  of  development  to  ensure 
that  it  meets  the  agency's  needs ^  followed  by  "freezing"  the 
products  of  that  phase,  will  allow  later  phases  to  proceed 
with  less  disruption.  Minor  changes  may  be  introduced  into 
the  system  maintenance  process  with  less  disruption  after  the 
system  is  implemented.  If  a  certain  phase  does  not  meet  the 
agency's  needs,  it  should  be  immediately  reviewed  and  altered 
where  necessary. 

Five  of  the  nine  cases  we  studied  did  not  properly 
review  each  phase  of  development.  Iii  our  first  case,  the 
agency  made  no  attempt  to  reevaluate  its  contracting  decision 
when  the  basic  assumptions  of  it  proved  wrong.  Also,  the 
agency  frequently  changed  the  contract's  requirements  as  they 
were  being  executed  and  failed  to  promptly  answer  the  con¬ 
tractor's  questions.  Even  if  the  contract  should  have  been 
continued  despite  the  invalid  original  assumptions,  the  fre¬ 
quent  changes  the  agency  required  as  the  contract  proceeded 
made  it  impossible  later  to  deny  payment  for  poor  performance. 

In  our  sixth  case,  the  agency  made  no  provisions  in  the  contract 
for  monitoring  the  contractor's  work  and  did  not  attempt  to 
formally  do  so.  The  agency  also  requested  several  changes  to 
input  and  output  data  formats  and  grossly  underestimated  their 
impact  on  the  work.  Since  it  was  not  monitoring  the  work, 
the  agency  was  not  aware  of  the  serious  problems  until 
delivery  time. 

In  contrast,  our  successful  fifth  case  displayed 
extremely  close  agency  management  during  contract  execution- — 
thorough  benchmark  tests  and  technical  reviews  were  conducted 
throughout  the  execution  at  the  contractor  site,  and  excellent 
communications  between  the  contractor  and  the  agency  were 
maintained.  As  explained  in  the  case  write-up  (see  app.  II), 
a  strong  project  officer  and  a  well-defined  problem  helped  a 
great  deal,  but  we  feel  that  the  close  management  during  exe¬ 
cution  would  have  been  significant  even  without  these  added 
factors. 

Agencies  do  not  adequately 

inspect  and  test  software 

Most  software  delivered  in  the  case  studies  was  of  poor 
quality.  The  poor  quality  was  evident  in  all  phases  of 
development.  One  means  of  obtaining  higher  quality  software 
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is  for  the  agency  to  require  the  contractor  to  maintain  a 
quality  assurance  function  to  address  the  following  areas: 

Working  task  Assures  that  procedures  are  in  effect 

for  subdividing  and  initiating  work, 
and  that  someone  is  assigned  the  respon¬ 
sibility  of  getting  the  work  done. 

Configuration  Controls  that  will  insure  that  system 
management  design  does  not  deviate  from  the  base 
specifications. 

Testing  Various  levels  of  tests  should  be 

identified  and  scheduled  to  insure  the 
quality  of  the  product  as  developnent 
progresses. 

The  program  design  should  be  reviewed 
and  evaluated  before  the  programming 
phase  begins  to  prevent  the  contractor 
from  having  to  reprogram  if  the  design 
is  changed. 

Documentation  Procedures  to  insure  delivery  of  up-to- 

date  documentation  so  the  agency  will 
know  how  to  run,  modify,  and  maintain 
the  software. 

The  contractor's  plan  should  be  documented  and  subject  to 
periodic  review  by  the  agency  throughout  the  life  of  the 
contract.  Although  in  this  instance  we  are  speaking  about 
the  quality  of  the  software  itself,  quality  assurance  must 
be  a  part  of  the  agency  management  process  directed  toward 
receiving  the  quality  of  software  specified  in  the  contract. 
The  military  is  now  using  this  technique  on  software  con¬ 
tracts.  Specific  examples  of  where  quality  assurance  would 
have  been  useful  are  noted  in  the  first  and  second  cases. 

Agencies  fail  to  establish  a  single 

focal  point  for  the  contractor 

Establishing  a  single  point  within  the  agency  for  contact 
with  the  contractor  will 

— shorten  communication  lines, 

— give  the  contractor  one  place  to  go  for  answers, 

— reduce  duplication  of  effort,  and 


Program 

design 
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—give  one  group  within  the  agency  an  overview  of  the 
whole  development  effort. 

The  agency  officials  in  this  position  should  be  author- 
XulfSse^Shetr^allto^uJ  W*eisu^ftSir?hose  organizations 

in  case  2,  we  found  that  (1)  in  fyears,  several  different 

project  officers  were  assigned  to 

ny  With  aQencv  officials  other  than  tne  proj 

ect  officer#  and  (3)  the  contractor  worked  with  seven  separate 

loLry  bS«aus.  This  situation  was  counter-productive,  the 

n^cid  to  inform  new  project  officers  was  continual,  and  the 

weakness  of  agency  contract  management  was  underscored. 

Miscellaneous  causes  observed 

The  following  causes  also  contributed  to  the  case  prob¬ 
lems  we  encountered: 

—Poor  work  done  by  the  contractor,  such  incorrect 
programming  (first,  second,  and  ninth  case  ). 

_ Excessive  turnover  of  both  contractor  and  agency 

staff  (second  case). 

_ Contractor  not  qualified  for  the  job  (ninth  case). 

—Agency  staff  not  qualified  to  do  their  part  of  the 
work  (second  case)# 

—Documentation  of  the  agency  procedures  to  be  auto¬ 
mated  was  not  available  (fourth  case). 

How  the  successful  case 
differed  from  the  others 

One  notable  exception  among  the  case  studies  was  case 
study  number  five.  This  case  showed  excellent  management 
of  the  contracted  work  by  a  diligent  and  competent  3 
officer.  Excellent  contractor  selection 

tional  specifications,  monitoring  procedures,  acceptance 
testing,  and  a  strong,  consistent  management  effort  were 
all  factors  in  the  successful  completion  of  this  software 
contract. 
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AGENCIES  AND  CONTRACTORS  AGREED 
ON  SOME  CAUSES  OF  PROBLEMS  AND 
WAYS  TO  IMPROVE  CONTRACTING 

Contractors  and  agency  representatives  who  answered  our 
questionnaire  often  agreed  on  the  causes  of  problems  and  on 
ways  to  improve  software  development  contracting. 

Three  of  the  four  problem  causes  identified  by  agencies 
as  most  important  were  similarly  identified  by  contractdrs — (1) 
requirements  were  changed  well  into  the  contract  performance 
period,  (2)  acceptance  testing  procedures  were  not  specified 
in  the  contract,  and  (3)  the  contract  did  not  stipulate 
what  constitutes  satisfactory  contractor  performance. 

Concerning  means  of  improvements,  contractors  and  agency 
representatives  considered  the  same  four  actions  to  be  taken 
by  agencies  as  most  important,  and  ranked  them  in  the  same 
order:  (1)  define  satisfactory  contractor  performance  in 

contracts,  (2)  include  specific  acceptance  testing  require¬ 
ments  in  contracts,  (3)  specify  exact  documentation  required 
in  contracts,  and  (4)  designate  someone  who  will  be  authorized 
to  decide  on  questions  which  arise. 

Both  groups  also  considered  the  same  four  actions  by 
contractors  as  most  important,  but  ranked  them  differently; 

(1)  request  agencies  to  specify  acceptance  testing  require¬ 
ments  on  contracts  (ranked  highest  by  both  groups),  (2)  leave 
the  same  staff  on  contracts  from  start  to  finish,  (3)  request 
agencies  to  specify  documentation  requirements  on  contracts, 
and  (4)  work  more  closely  with  agencies  before  contract 
award  to  clarify  the  contract  language. 
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CHAPTER  4 


CONCLUSIONS /  RECOMMENDATIONS,  AND 
AGENCY  COMMENTS 


CONCLUSIONS 

Federal  agencies'  contracting  for  custom  computer 
software  development  needs  improvement.  While  many  software 
development  contracts  are  successful,  too  many  deliver  noth¬ 
ing  or  deliver  something  that  costs  much  more  and  takes  much 
longer  than  originally  estimated.  Many  Federal  software 
development  contracts  in  the  general  management  category 
that  have  experienced  trouble  share  common  problems.  We  be¬ 
lieve  that  the  actions  we  recommend  could  improve  the  situa¬ 
tion  significantly. 

Many  Federal  software  development  contracts  in  the  gen¬ 
eral  management  ADP  category  have  serious  problems.  At  the 
root  of  those  problems  are  five  important  causes.  First,  a 
lack  of  guidance  is  common  to  nearly  all  such  contracts. 

The  central  agencies— 0MB,  NBS,  and  GSA— have  not  issued 
adequate  guidance  on  software  development  contracting,  and 
most  other  agencies  have  not  attempted  to  fill  that  gap  by 
publishing  specific  guidance  of  their  own.  The  second  cause 
is  the  failure  of  Federal  agencies'  top  management  both  to 
realize  the  difficulty  and  the  importance  of  software  devel¬ 
opment  contracting  and  to  commit  appropriate  management  re¬ 
sources  and  adequately  trained  project  officers  to  it.  The 
third  cause  is  the  tendency  of  Federal  agencies  to  commit 
themselves  to  software  development  contracts  before  user 
function,  testing,  and  performance  criteria  have  been  so 
defined  as  to  3ustify  their  degree  of  commitment. 

Such  inappropriate  levels  of  commitment  happen  because 
agencies  have  overly-optimistic  assessments  of  the  work  they 
do  before  contracting  and  because  agencies  do  not  divide  con¬ 
tracts  into  phases  so  that  they  can  cancel  the  later  phases 
if  earlier  ones  are  not  satisfactory.  Contractors  particu¬ 
larly  emphasized  that  agencies  contract  without  adequate 
test  data.  When  agencies  do  not  adequately  define  perform¬ 
ance,  they  have  no  hope  of  later  denying  payment  for  poor 
performance. 

The  fourth  cause  of  contracting  problems  results  when 
an  agency  does  not  manage  the  work  of  software  development 
contracts  adequately  during  its  execution.  Changes  to  the 
scope  of  work,  expeditious  handling  of  contractor  questions, 
and  inspecting  work  progress  are  all  handled  poorly. 
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Lastly,  serious  contracting  problems  can  arise  when 
agencies  do  not  adequately  inspect  and  test  software.  When 
a  contract  originally  is  inadequately  specified  and  then  is 
not  adequately  inspected  and  tested,  the  agency  not  only 
gets  bad  software  but  also  cannot  deny  payment  to  the  con¬ 
tractor. 

RECOMMENDATIONS 


We  recommend  that  the  Secretary  of  Commerce,  through  the 
National  Bureau  of  Standards,  and  the  Administrator  of  the 
General  Services  collaborate  to  issue  specific  guidance  to 
assist  Federal  agencies  in  recognizing  and  managing  the  unique 
factors  involved  in  contracting  for  custom  software  development. 
We  recommend  that  they  collaborate  because  both  technical  and 
procurement  considerations  are  involved.  The  following  areas 
should  be  covered  in  the  guidelines: 

Procurement  (primarily  GSA's  responsibility): 

— Internal  agency  management  practices  necessary  to 
properly  write,  manage,  and  monitor  software  devel¬ 
opment  contracts  in  the  general  management  category 
of  ADP. 

— Specific  instructions  on  how  to  tailor  software  de¬ 
velopment  contracts  to  the  agency's  stage  of  system 
development  at  the  time  the  contract  is  let. 

— Contract  stipulations  to  subdivide  the  software 
development  into  phases. 

— The  review  and  approval  procedures  to  be  followed 
by  the  agency  at  the  end  of  each  phase. 

— How  to  apply  contract  clauses  to  require  as  a  con¬ 
dition  of  payment,  and  to  inspect  for,  both  the  pro¬ 
curement  requirements  and  the  technical  specifica¬ 
tions  (see  below). 

— HOW  to  handle  changes  in  software  under  development 
with  the  least  disruption. 

—The  effective  use  of  contract  clauses  to  ensure  the 
agency's  ability  to  deny  payment  when  the  contractor 
does  not  perform. 

— How  to  ensure  that  the  contractor  follows  sound  system 
development  practices. 
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Technical  (primarily  KBS'  responsibility): 

—Means  of  specifying  and  quantifying 

including  programming  P'^®=tlces  standards  and  adhe 
ence  to  programming  language  standards,  we  realise 
tHat  thele  Ireas  are  still  changing,  but  w® 
that  a  number  of  common  sense  guidelines  exist  y 
whose  widespread  publication  by  MBS  would  help. 

-Means  of  specifying,  defining,  accept- 

ance  testing  requirements,  and  documentation 
standards# 

.—Means  of  defining  satisfactory  performance  by  the 
contractor. 

--Guidance  on  performance  specifications  to  be  included 
thS  SoStrLt  to  clarify  the  quality  requirements 
for  the  software. 

one  way  that  the  working-level  project  manager  could 

mny-A  snecific  quidance  would  be  for  NBS  and  GSA  to  worK 

Sir's 

models,  ^he  agencies  cou  situation  and  creating 

f -mSdS?ar-  contract  "ith  adequate  provisions  and  safeguards. 

Although  Ois  P'®'*®®  p^im^lily  recommended  as 

rird"‘to'agenirsi?tre“i;trac“nlnaSmei^.  Regardless 
Tt  ?ie  ietSlISJJgrused,  guidance  should  not  be  so  broad  and 
general  as  to  exclude  specific  procedures. 

We  also  recommend  that  Federal  agencies  which  contract 
extensively  for  software  development  train  project 

Illustrated  in  Sis  report  and  the  checklist  shown  in  appen- 
ttt  1  S  Sirwith  contracting  for  software  development. 


1/For  example,  forbidding  COBoii  programmers  to  «se  the 

ALTER  verb  because  it  makes  their  programs  more  difficult 
for  others  to  understand.^ 
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AGENCY  COMMENTS 


We  asked  the  General  Services  Administration  and  the 
Department  of  Commerce  to  comment  on  our  draft  report. 

Their  replies,  which  indicated  general  agreement,  are  in¬ 
cluded  as  appendix  III  to  this  report  and  are  discussed  below. 

The  Administrator  of  General  Services  agreed  with  our 
conclusions  and  recommendations  and  stated  that  the  Auto¬ 
mated  Data  and  Telecommunications  Service  of  GSA  would  wel¬ 
come  the  opportunity  to  collaborate  with  NBS  in  carrying 
out  the  recommendations. 

Addressing  our  recommendation  that  NBS  and  GSA  collab¬ 
orate  to  issue  guidance  to  Federal  agencies,  the  Administra¬ 
tor  said  that  the  Automated  Data  and  Telecommunications  Ser¬ 
vice  is  developing  contracting  guidance  for  it  own  use  in 
providing  services  to  other  Federal  agencies  and  that  the 
Service  welcomes  the  opportunity  to  collaborate  with  NBS  on 
extending  this  guidance  to  other  agencies. 

Concerning  our  suggestion  that  guidance  could  be  pro¬ 
vided  with  model  contracts,  he  said  that  the  Service  is  also 
developing  standard  terms  and  conditions  for  software  devel¬ 
opment  contracting  as  part  of  its  contract  services  program 
and  would  like  to  collaborate  with  NBS  in  this  effort. 

With  respect  to  our  recommendation  that  Federal  agen¬ 
cies  train  project  managers,  he  said  that  the  Automated  Data 
and  Telecommunications  Service  either  provides  training  or 
lends  project  managers  on  a  reimbursable  basis. 

The  Department  of  Commerce  generally  concurred  with  our 
recommendations.  Both  the  Assistant  Secretary  for  Science 
and  Technology  and  the  Department's  Office  of  Procurement 
and  ADP  Management  provided  comments.  However,  neither  made 
any  reference  to  collaborating  with  GSA  on  governmentwide 
guidance. 

Concerning  our  recommendation  that  Federal  agencies 
train  project  managers  and  ensure  that  appropriate  contract¬ 
ing  action  is  taken,  the  Commerce  Department's  Office  of 
Procurement  and  ADP  Management  said  that  it  is  planning  a 
series  of  procurement  briefings  and  is  distributing  our  pro¬ 
visional  checklist  throughout  the  Department.  The  Office 
also  indicated  that  it  recently  published  guidelines  for  the 
preparation  of  technical  packages  for  ADP  solicitations.  A 
Commerce  official  told  us  that  these  actions  are  for  use  within 
the  Department  only. 
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With  regard  to  our  recommendation  that  NBS  collaborate 
with  GSA  in  developing  governmentwide  guidelines,  the  Assis¬ 
tant  Commerce  Secretary  for  Science  and  Technology  stated 
that  StioLl  guidelines  for  the  unique  problems  of  con¬ 
tracting  for  custom  software  are  not  in  the  P^®5  oi-c 

app^oJed  by  the  Congress.  He  stated  that  other  NBS  products, 
including  programming  language  standards  and  documentation 
guidelines,  could  be  applied  to  software  con¬ 

tractors,  as  well  as  to  software  developed  in-house.  He 
also  forwarded  NBS'  specific  comments.  (See  app.  HI*)  Spe 
events  1,  2,  4,  5,  6,  and  7  were  incorporated  into 
this  final  report  almost  exactly  as  NBS  sent  them. 

concerning  the  agency's  third  °''SbS 

wording  to  follow  more  closely  the  wording  of  an  earlier  NBS 

response  to  our  inquiry  about  the  scope  of  J^s  ^2 

response  to  NBS'  eighth  comment,  we  ^«^®  /se^ 

nosal  evaluation  phase  to  our  provisional  checklist  vo©© 
app.  1),  which  inlludes  the  elements  HBS 

the  added  phase  was  not  "developed  extensively  as  NBS  sug 

gests  because  extensive  development  ?2%aiied” 

mending  that  NBS  do— that  is  why  our  checklist  is  called 

provisional. 
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APPENDIX  I 


PROVISIONAL  CHECKLIST  FOR 
SOFTWARE  DEVELOPMENT  CONTRACTING 

This  checklist/  which  we  prepared  during  our  review/ 
lists  matters  which  we  feel  agency  management  should  consider 
when  contracting  for  computer  software  develojwient .  The 
checklist  is  divided  into  five  contract  phases:  pre-contract/ 
contract/  proposal  evaluation/  performance  period/  and  post¬ 
contract. 

While  this  checklist  is  only  an  interim  document/  we 
feel  that  it  will  be  useful  to  persons  involved  in  contracting 
for  software  development. 

We  are  aware  that  some  of  the  points  included  in  this 
list  might  apply  to  contracting  for  any  type  of  service  and 
may  be  covered  in  general  procurement  guidance.  We  list 
those  points  here  to  make  the  list  complete  and  to  emphasize 
that  they  can  indeed  be  applied  to  software  developnent  con¬ 
tracts  as  well  as  to  others. 
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Pre-contract 

To  obtain  realistic  assessments 
of  the  size,  scope,  and  cost  of 
the  effort  to  produce  the  needed 
software. 

METHODS; 

— Describe  the  needed  software  as  completely  as  possible 
in  the  request  for  proposal  so  that  the  contractor  can 
understand  and  address  the  full  scope  of  work  in  his 
proposal. 

--Describe  the  software  so  that  people  not  familiar  with 
agency  operations  can  understand  the  need. 

— Develop  criteria  to  evaluate  how  reasonable  and  accurate 
the  contractor's  costs  are. 

PROCEDURES ; 

—include  all  details  from  the  system  development  steps 
completed  by  the  agency  to  date. 

— In  the  areas  where  detailed  specifications  have  not 
been  developed,  clearly  state  the  functional  require¬ 
ments  the  software  must  satisfy. 

— Give  all  known  constraints  and  parameters  the  vendor 
must  work  with  in  developing  the  software. 

POSSIBLE  WAYS  OF  ACCOMPLISHING 

THE  PROCEDURES; 

—The  use  of  agency  jargon,  which  might  be  unclear  to 
outsiders,  should  be  avoided. 

—A  contractor's  accounting  system  must  be  determined 
to  be  adequate  to  generate  valid  cost  estimates. 

—The  cost  of  past  development  efforts  may  be  compared 
to  current  proposed  costs. 

—Caution  should  be  exercised  on  bids  that  are  much 
higher  or  lower  than  the  average  of  bids  received. 


PHASE; 

OBJECTIVES; 
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PHASE :  Pre-contract  (cont.) 

OBJECTIVE:  To  insure  the  contractor  most  suited 

to  do  the  work  is  selected 

METHOD; 

Develop  comprehensive  proposal  evaluation  standards  to 
use  in  identifying  nonresponsive  bidders  and  in  select¬ 
ing  a  qualified  contractor. 

PROCEDURES ; 

— Consider  the  contractor’s  management  qualifications. 

1.  Does  the  contractor's  organization  reflect 
adequate  management  overall? 

2.  Is  adequate  management  available  for  this  proj¬ 
ect? 

3.  Will  the  staff  responsible  for  the  proposal 
also  work  on  the  project? 

4.  Is  the  responsibility  for  this  project  at  a 
high  enough  level  of  authority  in  the  contrac¬ 
tor's  organization? 

5.  Is  it  likely  that  key  personnel  will  remain 
with  the  project  from  start  to  finish? 

— Consider  the  contractor's  technical  approach. 

1.  Does  it  reflect  a  knowledge  of  the  agency's 
mission  and  functions? 

2.  Will  the  proposed  software  take  full  advantage 
of  the  agency's  latest  computer  hardware 
capability? 

3.  Does  the  contractor  use  software  performance 
measurement  tools  and  techniques  as  well  as 
software  optimization  tools  to  insure  the 
most  efficient  development  possible? 

4.  Does  the  technical  proposal  rely  on  state  of 
the  art  or  unproven  methodology  as  opposed 
to  proven  technology? 

5.  Is  the  overall  design  sound  and  feasible? 
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— Consider  the  contractor's  quality  assurance  program. 

1.  Does  the  contractor  have  a  documented  program 
in  effect  that  will  ensure  that  the  software 
meets  contract  specifications? 

2.  Is  contractor  quality  assurance  measurement 
compatible  with  the  agency  acceptance  criteria 
for  the  final  product? 

— Consider  the  contractor's  cost  proposal. 

In  doing  so,  refer  to  the  criteria  in  the 
first  objective  of  this  phase. 
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PHASE; 

OBJECTIVE; 

METHOD; 


To  initiate  a  framework  for  the 
software  development  that  ensures 
the  maximum  chance  for  success  by 
requiring  that  sound  system  devel 
opment  practices  be  followed. 


Include  iri  the  contract  a  requirement  that  the  contrac¬ 
tor  not  bypass  system  development  steps  in  the  develop¬ 
ment  process.  This  will  provide  a  means  for  the  agency 
to  assure  itself  that  the  contractor  is  performing  the 
required  system  development  steps. 

PROCEDURES ; 

— Determine  the  system  development  steps  accomplished 
by  the  agency  to  date. 

— Identify  the  remaining  system  development  steps  to  be 
accomplished  by  the  contractor. 

—Specify  the  sequence  in  which  these  steps  are  to  be 
accomplished. 

POSSIBLE  WAY  OF  ACCOMPLISHING 

THE  PROCEDURES; 

As  part  of  the  deliverables  that  the  contractor  must 
provide,  include  a  product  at  the  end  of  each  system 
development  step  which  demonstrates  that  the  step  has 
been  satisfactorily  completed.  These  products  would 
include  preliminary  surveys,  feasibility  studies, 
general  and  detail  design,  test  data  and  test  plan, 
the  actual  programs,  and  acceptance  test  results. 
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PHASE: 


Contracting  (cont.) 


OBJECTIVE: 


Write  a  contract  which  clearly  states 
the  obligations  of  both  the  agency  and 
the  contractor. 


METHODS: 

—Specify  in  the  contract's  statement  of  work  who  has 
the  responsibility  for  each  task. 

—Specify  in  the  contract  what  constitutes  satisfactory 
performance  by  the  contractor. 


PROCEDURES: 


—All  agency  obligations  should  be  included  in  the 
milestone  chart  and  their  performance  period  should 
be  specified  the  same  as  the  contractor  s. 

—Tasks  should  be  scheduled  so  that  agency  work  that  is 
prerequisite  to  contractor  performance  will  not  delay 
the  contractor. 

—The  contract  should  provide  for  phasing  the  software 
from  the  contractor  to  the  agency,  including  agency 
tuning  and  briefing  the  staff  during  the  work. 


POSSIBLE  WAYS  OF  ACCOMPLISHING 
THE  PROCEDURES: 


Whether  the  specifics  of  the  software  are  known  or 
not,  satisfactory  performance  should  be  quantified  in 
terms  of  all  known  requirements  and  constraints.  Some 
areas  for  performance  standards  are  the 

_ ability  to  meet  agency's  functional  requirements. 


— growth  potential  requirements  of  the  system, 
—time  constraints  for  deliverables. 


_ test  and  acceptance  criteria  which  must  be  met, 

— programming  language  standards  and  practices 
standards  to  be  followed, 

— documentation  standards  to  be  followed, 

— ease  of  modification,  and 


— maximum  computer  resources  allowed. 
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PHASE:  Contracting  (cont.) 

OBJECTIVE :  Write  a  contract  which  gives  the  agency 

the  right  and  the  visibility  to  manage 
the  contract. 

METHODS: 

—Provide  for  a  minimum  of  investment  of  funds  before 
the  quality  of  the  contractor's  work  is  known. 

Specify  who  is  authorized  to  make  changes  in  the 
contract. 

Provide  some  means  to  monitor  the  contractor's  progress. 
PROCEDURES : 

—Phase  the  development  into  logical  work  steps. 

—Require  agency  approval  for  each  phase  before  begin¬ 
ning  work  on  the  next. 

The  more  undefined  the  software  is^  the  closer  the 
phasing  steps  should  be  at  the  outset. 

POSSIBLE  WAYS  OF  ACCOMPLISHING 
THE  PROCEDURES: 

—Besides  the  contracting  officer,  specify  management's 
technical  representative  who  is  authorized  to  make 
changes  and  to  answer  contractor's  questions. 

Use  phasing  to  assist  in  setting  up  a  milestone  chart 
showing  the  time  frame  for  each  work  step. 

—Require  the  contractor  to  document  the  satisfactory 
completion  of  each  workstep  by  the  agreed  date. 

Interim  progress  reports  may  be  required  from  the 
contractor  to  allow  progress  to  be  monitored  during 
each  work  step. 

Identify  performance  as  well  as  functional  speci¬ 
fications. 

Specify  the  measures  of  reliability  and  quality  by 
which  the  contractor's  work  will  be  evaluated. 
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PHASE:  Contracting  (cont») 

OBJECTIVE:  Write  a  contract  which ^giyes  the 

agency  some  recourse  if  the  contrac¬ 
tor  fails  to  perform. 

METHODS : 

—The  contract  should  provide  for  the  agency  to  termi¬ 
nate  the  contract  if  it  becomes  evident  that  the  con¬ 
tractor  cannot  perform  according  to  the  contract's 
terms . 

— The  contract  should  also  provide  separate  due  dates 
and  costs  for  each  deliverable  and  a  provision  to 
withhold  payment  for  incomplete  or  unacceptable  work. 

PROCEDURE : 

If  agency  evaluation  of  the  contractor's  work  indi¬ 
cates  that  the  approach  being  used  will  not  produce 
the  needed  software,  and  if  the  contractor's  work 
cannot  be  redirected,  the  contract  should  be  termi¬ 
nated  to  minimize  losses. 

POSSIBLE  WAYS  OF  ACCOMPLISHING 

THE  PROCEDURE:  ~  ” 

—Satisfactory  performance  criteria,  and  acceptance 
testing  criteria  should  be  used  to  identify  work  that 
does  not  meet  contract  requirements. 

— Payment  to  the  contractor  should  be  reduced  by  the 
amount  of  any  deliverables  (e.g.,  documentation) 
specified  in  the  contract  but  not  produced. 
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PHASE; 

OBJECTIVE: 


METHOD: 


Proposal  evaluation 

Ensure  that  a  skilled  and  responsible 
contractor  is  selected. 


Only  contractors  with  verified  technical  skill  and 
financial  responsibility  should  be  selected  to  develop 
custom  software  contracts. 


PROCEDURES : 


Agencies  should  evaluate  each  proposal  in  terms  of 
relevant  factors,  including  those  listed  below,  and 
including  considerations  of  feasibility,  comparison  to 
other  proposals'  prices  and  schedules,  and,  if  possible, 
comments  solicited  from  the  contractor's  prior  customers. 


POSSIBLE  WAYS  OF  ACCOMPLISHING 
THE  PROCEDURES; 

— Include  in  the  request  for  proposal  a  provision  for 
agency  representatives  to  visit  the  contractor  site 
to  investigate  and  evaluate  various  factors,  including 
finanical  position,  technical  capability,  and  labor 
resources. 

--Try  to  determine  whether  or  not  the  contractor's 

capabilities,  qualifications,  and  experience  are  Com¬ 
mensurate  with  the  complexity  of  the  problem. 

— Try  to  determine  whether  or  not  the  contractor's 
staff  has  experience  with  the  software  and  hardware 
to  be  used  during  development. 


— Consider  the  effect  on  the  contract  schedule  of  the 
contractor's  experience  (e.g.,  delay  in  training  the 
contractor's  staff). 
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PHASE: 


Performance  period 


OBJECTIVES:  The  agency  should  manage  the  contract 

during  execution  in  a  manner  that 
contributes  to  its  success. 

METHODS: 


— The  agency  should  provide  all  of  its  deliverables  to 
the  contractor  within  the  specified  time  frames. 

— Management  should  create  an  environment  within  the 
agency  that  supports  the  contractor's  efforts. 

PROCEDURES : 

—Such  things  as  studies,  requirements,  and  general  de¬ 
signs,  to  be  provided  by  the  agency  should  be  pro¬ 
vided  promptly  so  that  the  contractor  is  not  delayed. 

— If  provided  by  the  agency,  such  work  products  should 
be  complete  and  accurate  and  provide  a  basis  for  the 
contractor's  work. 

POSSIBLE  WAYS  OF  ACCOMPLISHING 

THE  PROCEDURES: 

— Management  should  be  involved  at  high  enough  levels 
to  insure  cooperation  by  all  agency  personnel.  If 
there  is  disagreement  about  the  new  development  effort 
among  various  factions  within  the  agency,  it  should 
be  resolved  in-house  by  management  and  not  left  for 
the  contractor  to  encounter. 

— If  possible,  the  same  agency  staff  should  be  kept  on 
the  project  throughout  the  contract. 

— A  single  focal  point  should  be  established  within  the 
agency  to  deal  with  the  contractor. 

— An  open  line  of  communication  should  be  maintained 
with  the  contractor.  However,  undocumented  informal 
communication  can  lead  to  cost  overruns.  Any  changes 
in  the  scope  of  work  must  be  handled  by  formal  modi¬ 
fication  of  the  contract  document. 
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PHASE ;  Performance  period  (cont.) 

OBJECTIVE ?  Monitoring  the  contractor's  progress 

to  ensure  that  critical  milestones  are 
met  and  to  approve  work  segments. 

METHODS; 

— Use  time  frame  criteria  that  are  established  to  de¬ 
termine  whether  the  contractor's  development  is  on 
schedule. 

— All  work  should  be  reviewed  at  the  end  of  each  com¬ 
pleted  phase  to  determine  if  it  conforms  with  con¬ 
tract  specifications. 

PROCEDURES ; 

“—Check  progress  against  dates  on  the  milestone. 

—Be  alert  for  signs  of  problems  in  the  contractor's 
progress  reports. 

Conduct  any  independent  reviews  of  the  contractor's 
operation  that  are  considered  necessary. 

POSSIBLE  WAYS  OF  ACCOMPLISHING 
THE  PROCEDURES; 

Consider  how  well  agency  requirements  are  met. 

— Decide  if  the  contractor's  approach  is  technically 
feasible. 

— Obtain  users'  evaluation  of  contractor's  work.. 

— Render  timely  management  decisions  on  all  alternatives 
presented  by  the  contractor. 

—Once  a  phase  is  approved,  freeze  that  phase  until  de¬ 
velopment  is  complete  to  stabilize  the  base  for  suc¬ 
ceeding  phases. 

Acceptance  testing  may  be  applied  to  completed  phases 
as  well  as  at  the  end  of  the  development  effort. 
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PHASE: 


Performance  period  (cont.) 


OBJECTIVE: 


Do  adequate  testing  to  ensure  that  the 
product  meets  contract  specifications. 


METHOD  I: 

— 'Make  sure  acceptance  test  criteria  are  still  current. 
— Maintain  adequate  control  over  the  testing  operation. 
PROCEDURE: 

If  test  criteria  and  data  were  developed  in  the  begin¬ 
ning,  make  certain  they  have  been  revised  to  incorpor¬ 
ate  system  changes,  if  any. 

POSSIBLE  WAYS  OF  ACCOMPLISHING 
THE  PROCEDURE:  ~ 

—Observe  or  participate  in  software  test. 

—Adequately  analyze  the  test  results. 

• — Document  all  errors  revealed  by  the  test. 

—Require  the  contractor  to  correct  all  discrepancies 
as  a  condition  for  final  payment. 

— Follow  up  on  all  discrepancies  to  make  sure  they  are 
corrected  before  the  system  is  accepted. 


METHOD  II: 

Assure  that  personnel  responsible  for  testing  the  sys¬ 
tem  have  adequate  technical  expertise. 

PROCEDURES: 

—Assign  qualified  agency  staff  with  systems,  data  pro¬ 
cessing,  and  performance  evaluation  expertise  to  test 
software. 

—If  the  agency  does  not  have  the  expertise  to  ade¬ 
quately  evaluate  the  software,  arrange  for  an  inde¬ 
pendent  evaluation  by  outside  sources. 
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Performance  period  (cont.) 

To  exercise  all  remedies  on  behalf  of 
the  agency  in  case  the  contractor  fails 
to  perform. 


Full  payment  should  not  be  made  to  the  contractor  until 
It  IS  determined  that  the  software  meets  contract  spec¬ 
ifications.  ^ 

PROCEDURE; 


Exercise  the  contract  recourse  provisions  discussed 
under  contract  controls  to  the  degree  that  nonperform¬ 
ance  was  encountered. 


APPENDIX  I 

PHASE; 

OBJECTIVE; 

METHOD ; 
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PHASE :  Post-contract 

OBJECTIVE:  Identify  both  good  and  bad 

-  aspects  of  the  software  procurement. 

METHOD: 

Make  a  followup  analysis  of  the  software  development 

procurement  contract  to  evaluate  contracting  f>ractices, 

record  lessons  learned,  and  evaluate  user  satisfaction 

with  the  product. 

PROCEDURES : 

— Identify  practices  that  are  weak  and  need  to  be 
changed . 

—Identify  and  retain  practices  that  produced  good 
results. 

— Identify  additional  agency  guidelines  that  need  to  be 
developed  and  implemented. 

— Evaluate  user  satisfaction  with  the  software. 

— Record  the  actual  amount  of  maintenance  programming 
work  that  is  needed  soon  after  the  software  is  put 
into  production. 

—Retain  performance  data  on  the  individual  contractor 
for  future  reference. 
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CASE  STUDIES  IN  CONTRACTING 
FOR  COMPUTER  SOFTWARE  DEVELOPMENT 

Following  are  summaries  of  the  cases  we  examined  in 
detail  during  our  review.  Typically,  several  causes  contri¬ 
buted  to  the  problems  in  each  case.  Several  of  these  cases 
came  to  our  attention  because  they  were  known  to  be  problem 
situations,  and  therefore,  these  cases  cannot  be  taken  as 
representative  of  the  majority  of  Federal  software  develop¬ 
ment  contracts. 

However,  we  offer  these  case  descriptions  to  illustrate 
some  of  the  things  that  can  go  wrong— and  some  of  the  things 
that  can  be  done  right— because  we  feel  that  others  may  be 
helped  by  the  experience  described. 
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Case  Study  1  -  The  payroll  system  that  never  paid  anyone 

An  agency  contracted  for  the  design,  programming, 
testing,  and  documentation  of  a  payroll  system  for  its  head¬ 
quarters,  regions,  and  service  support  locations.  The  cost 
and  time  were  originally  estimated  at  about  $300,000  and  1 
year.  In  the  end,  the  contract  cost  $1  million  and  lasted 
4  years.  Furthermore,  the  agency  found  the  delivered  system 
unsatisfactory  and  eventually  abandoned  efforts  to  make  it 
work . 

What  happened  during 
the  contract 


It  appears  that  the  agency  made  the  following  assump¬ 
tions  when  contracting  for  the  system; 

— The  user  requirements  and  loose  general  design  devel¬ 
oped  by  the  agency  would  allow  the  contractor  to  pro¬ 
ceed  with  the  detailed  design  phase. 

— An  existing  payroll  system  developed  in-house  could 
be  adapted  to  meet  agency  requirements. 

—Based  on  the  above,  the  payroll  system  would  cost 
about  $300,000  and  would  take  about  1  year  to  develop. 

The  contract  called  for  doing  the  work  in  three  phases: 
system  organization  and  design;  detailed  logic  design  and 
program  specifications;  and  programming,  testing,  and  docu¬ 
mentation.  The  agency  was  to  approve  each  phase  before  the 
contractor  could  begin  work  on  the  next.  Satisfactory  per¬ 
formance  was  defined  only  in  general  terms  and  no  provision 
was  made  for  denying  payment  for  poor  performance  or  lack  of 
performance  by  the  contractor. 

Acceptance  test  procedures  were  not  clearly  defined; 
the  contractor  was  to  provide  test  specifications  that  would 
allow  the  agency  to  test  the  system.  Although  the  contractor 
could  suggest  changes,  only  the  agency  could  initiate  them. 

The  programming  language  was  specified,  and  documentation  was 
required  to  meet  standards  published  by  the  agency.  The  agency 
was  to  designate  one  employee  to  serve  as  the  official  repre¬ 
sentative.  No  subcontractors  were  involved. 

The  contractor's  project  manager  stated  that  two  things 
became  evident  early  in  the  contract — the  user  requirements 
developed  by  the  agency  were  useless  and  modifying  the  earlier 
payroll  would  not  meet  the  agency's  needs.  As  a  result,  the 
contractor  had  to  develop  a  valid  set  of  requirements. 
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Thus,  the  assumptions  and  estimates  on  which  the  con¬ 
tract  was  based  became  invalid  after  about  9  months'  work: 

(1)  the  initial  user  requirements  and  system  concept  were 
not  usable  because  the  system  concept  changed  from  decentral¬ 
ized  to  centralized,  (2)  modifying  the  existing  system  was 
no  longer  feasible  because  it  was  a  small-scale,  poorly- 
organized,  second-generation  system,  (3)  because  the  first 
two  assumptions  proved  invalid,  the  original  time  and  cost 
estimates  had  to  be  revised. 

Besides  the  change  in  the  system's  concept,  the  agency 
made  many  other  changes  during  the  contract.  Many  of  the 
changes  were  made  to  phases  already  approved,  and  after  work 
had  begun  on  succeeding  phases.  The  changes  were  additional 
requirements  and  changes  in  design  specifications;  one  was 
estimated  to  have  taken  about  6  months.  The  contractor  had 
to  ask  for  a  freeze  on  all  system  changes  several  times  be¬ 
fore  the  agency  finally  agreed  near  the  end  of  the  contract. 

The  contractor  stated  that  he  had  difficulty  getting 
agency  management  to  make  timely  decisions  on  critical 
issues  and  that  this  difficulty  contributed  to  the  overrun. 

At  one  point  the  contractor  tried  to  get  the  agency  to  agree 
to  a  consolidated  schedule  for  the  completion  of  contract 
actions  by  both  the  contractor  and  the  agency.  The  schedule 
was  never  agreed  to. 

After  about  4  years,  the  agency  brought  the  system  in- 
house  to  finish  and  install.  In  testing  it,  the  agency  made 
several  observations.  The  payroll  system  did  not  perform  as 
expected  1/  for  the  agency's  payroll  department  in  retirement, 
payroll  distribution  and  projection,  late  and  amended  time 
and  attendance  sheets,  or  Fair  Labor  Standards  Act  require¬ 
ments.  Also,  the  system  could  not  pay  the  required  number 
of  employees  in  the  2-week  pay  cycle. 

Besides  its  deficiencies  in  user  terms,  the  system  had 
technical  deficiencies:  (1)  it  had  18  unexpected  failures 
to  continue  processing  2/  during  the  test,  (2)  it  required 
excessive  computer  time  for  processing,  (3)  it  required  ex¬ 
cessive  intervention  by  programmers  to  keep  it  running,  (4) 
the  programs  in  the  system  contained  faulty  logic  and  ques¬ 
tionable  programming  techniques,  (5)  the  documentation  was 


1/In  ADP  terms,  it  "did  not  automate  these  user  functions. 
VCalled  "crashes"  in  ADP  terms. 
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not  detailed  enough,  and  (6)  the  old  decentralized  punched- 
card-input  concepts  were  perpetuated  throughout  the  system. 
The  agency  finally  abandoned  its  efforts  to  make  the  system 
work. 

Why  this  contract  failed 

The  contract  omitted  important  provisions  or  only  men¬ 
tioned  them  in  general  terms — satisfactory  performance  was 
not  defined,  the  agency  had  no  recourse  in  the  case  of  un¬ 
satisfactory  performance,  no  acceptance  testing  procedures 
were  established,  and  software  quality  assurance  was  not  re¬ 
quired  by  the  contractor. 

The  agency  incorrectly  assessed  its  system  development 
status  at  the  time  the  contract  was  let.  The  overoptimistic, 
incorrect  assessment  had  these  effects: 

— When  the  contract  assumptions  became  invalid,  the 
contractor's  scope  of  work  increased,  thus  increasing 
the  contract's  length  and  cost. 

— The  attempt  to  adapt  work  done  under  the  old  concept 
for  use  with  the  new  concept  caused  problems  in  the 
new  system.  The  fact  that  the  computer  programs 
still  reflected  the  old  concepts  contributed  strongly 
to  the  excessive  complexity  and  unsatisfactory  perfor¬ 
mance  of  the  new  system. 

- — The  agency's  pre-contract  work  was  wasted  because  the 
contractor  had  to  re-do  it. 

The  agency  apparently  did  little  to  verify  that  the  ex¬ 
isting  payroll  system  could  be  adapted,  and  we  have  no  indi¬ 
cation  that  it  reassessed  costs  when  this  proved  impossible. 
The  increased  work  to  design  a  completely  new  system  not  only 
increased  overall  costs,  but  may  have  eliminated  the  cost 
advantage  which  was  used  to  justify  the  contract. 

Besides  increasing  time  and  costs,  the  excessive  changes 
initiated  by  the  agency  had  secondary  effects. 

— Much  of  the  advantage  of  phasing  that  the  contract 
called  for  was  lost  because  the  agency  made  changes 
in  previously  approved  and  completed  phases  ex¬ 

ample,  the  formerly  approved  requirements  definition 
was  changed  after  programming  started).  The  changes 
destroyed  the  approved  work,  schedule,  and  quality 
control  of  the  earlier  phases.  Hence,  the  agency 
lost  much  of  its  ability  to  control  and  monitor  the 
direction  of  the  system's  development. 
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—The  agency  complained  that  the  program  documentation 
did  not  match  the  programs;  the  large  number  of 
changes  undoubtedly  contributed  to  this  situation. 

The  frequent  attempts  by  the  contractor  to  get  the  agency  to 
freeze"  the  system  give  some  indication  of  the  problems  the 
changes  caused. 

Agency  management's  failure  to  make  timely  decisions 
certainly  contributed  to  the  contract  overrun.  In  an  over¬ 
all  schedule,  agency  management  did  not  list  its  own  areas 
of  responsibility  as  it  had  those  of  the  contractor.  Man¬ 
agement  was  not  strong  enough  to  make  the  agency's  subdivi¬ 
sions  follow  the  overall  plan  so  as  not  to  delay  the  con¬ 
tract.  This  internal  disagreement  not  only  delayed  the 
contract,  but  also  made  it  impossible  for  the  agency  to  hold 
the  contractor  accountable  for  his  responsibilities. 

Even  if  the  clauses  were  included  in  the  contract  to 
allow  the  agency  to  deny  payment  for  unsatisfactory  perfor¬ 
mance,  they  would  have  been  useless  because  the  agency  con¬ 
tributed  to  the  causes  of  inadequate  contractor  performance. 
In  this  case  study,  if  the  contract  had  contained  such 
clauses  and  if  the  agency  had  not  delayed  the  contract,  the 
agency  apparently  would  have  been  justified  in  withholding 
payment  because  of  the  poor  product  the  contractor  delivered 
and  the  lack  of  documentation. 

The  poor  overall  quality  of  the  product  indicated  that 
the  agency  should  have  included  a  contract  provision  for  a 
contractor  software  quality  assurance  program  that  would 
detect,  analyze,  and  correct  deficiencies  in  the  software 
during  development. 
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Case  Study  2  -  A  $2-million,  contractor-developed  management 
information  system  replaced  by  a  $20,000 
in-house  version 

In  this  case,  an  agency  contracted  for  a  management 
information  system  that  would  allow  it  to  monitor  activities 
in  its  seven  bureaus.  The  contract  called  for  the  Govern¬ 
ment's  liability  to  be  limited  to  $378,147  and  the  work  to 
be  done  in  about  9  months.  The  contract  actually  cost 
$2,200,000  and  lasted  44  months,  during  which  time  it  was 
modified  16  times.  Despite  the  extra  time  and  funds  spent, 
the  agency  could  never  use  the  system  and  finally  abandoned 
attempts  to  do  so.  Agency  employees  later  developed  a  much 
less  sophisticated  system  which  met  the  basic  requirements 
and  cost  about  $20,000. 

What  happened  during  the  contract 

In  an  earlier  contract,  the  contractor  had  developed  a 
small  system  for  one  of  the  agency  bureaus.  A  contract  mod¬ 
ification  asked  the  contractor  to  expand  that  system  to  be 
used  agencywide.  Under  this  modification,  the  contractor 
identified  the  objectives  and  developed  a  general  implemen¬ 
tation  plan  for  the  larger  system.  Parts  of  the  first  sys¬ 
tem  were  to  be  adapted  to  the  new  one.  Because  of  the  pre¬ 
liminary  work  done  under  the  modification  to  the  earlier 
contract,  and  because  it  was  believed  that  part  of  an  exist¬ 
ing  system  could  be  used,  the  agency  perceived  itself  to  be 
well  into  the  system  development  cycle  when  this  contract  was 
let. 

The  cost-plus-fixed-fee  contract  was  awarded  as  a  sole- 
source  contract  because  of  the  contractor's  experience  and 
knowledge  of  the  system.  The  work  to  be  done  was  based  on 
two  documents  submitted  by  the  contractor  in  the  original 
proposal:  (1)  the  conceptual  design  which  specified  system 

requirements  and  the  system  description  and  (2)  the  imple¬ 
mentation  plan  which  outlined  the  tasks  necessary  to  accom¬ 
plish  detailed  design  and  system  development. 

The  contract  was  faulty  from  the  outset.  The  contract 
specified  that  inferior  and  unsatisfactory  workmanship  would 
be  corrected  at  no  extra  charge;  however,  the  contract  did 
not  specifically  define  satisfactory  performance  or  specify 
acceptance  testing  procedures.  The  implementation  plan  called 
for  monthly  progress  reports,  but  the  contract  did  not  con¬ 
tain  documentation  standards;  they  were  prepared  by  the  con¬ 
tractor  after  the  contract  was  awarded.  The  statement  of 
work  did  not  require  the  contractor  to  provide  any  evidence 
of  a  quality  assurance  program. 
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The  problems  of  this  contract  were  many.  First,  the 
people  who  had  worked  on  the  system  previously  developed  for 
the  agency  no  longer  worked  for  the  contractor.  It  was  their 
knowledge  and  experience  that  originally  prompted  the  agency 
to  award  the  contract  to  the  contractor  on  a  sole-source 
basis.  Second,  the  contractor  notified  the  agency  soon  after 
the  contract  was  awarded  that  the  programs  developed  for  the 
earlier  system  could  not  be  adapted  to  the  new  system. 

The  agency  memorandum  recommending  award  of  the  con¬ 
tract  stated  that  the  agency  bureaus  had  reviewed  the  con¬ 
ceptual  design  submitted  by  the  contractor  and  agreed  that 
it  met  their  requirements.  Certain  bureau  representatives, 
however,  disagreed.  It  was  under  these  conditions  that  the 
contract  effort  began. 

From  the  outset  the  contractor  experienced  difficulty 
in  delivering  work  on  time  to  meet  agency  approval.  The 
contractor  had  originally  proposed  to  complete  an  agency- 
approved  system  design  in  6  months.  To  accomplish  this,  the 
contractor  had  to  promptly  submit  documents  describing  the 
system  design  and  operation  to  the  agency  for  review.  After 
6  months,  the  contractor  still  had  not  submitted  all  the 
documents  for  approval,  and  none  of  those  that  had  been  sub¬ 
mitted  was  completely  satisfactory  to  the  agency.  After  6 
more  months,  the  agency  approved  the  system  design  but  con¬ 
tinued  to  ask  the  contractor  to  change  it.  The  contractor 
advised  the  agency  that  even  small  changes  could  seriously 
affect  the  work  to  be  done. 

After  the  design  was  approved,  the  contractor's  effort 
shifted  to  system  implementation,  which  was  to  be  completed 
in  4  months.  After  3  months,  the  implementation  had  begun 
in  only  two  of  the  seven  bureaus.  The  contractor  then  pro¬ 
posed  to  continue  implementation  work  for  16  more  months  at 
at  an  added  cost  of  $700,000.  The  proposal  also  expanded  the 
contractor's  role  in  helping  the  agency  control  system  oper¬ 
ations.  The  agency  approved  $360,000  of  this  amount  and 
agreed  to  develop  common  definitions,  categories,  and  proce¬ 
dures  for  assigning  descriptive  data  to  like  transactions  in 
all  seven  bureaus.  This  task  proved  too  technical  and  diffi¬ 
cult  for  agency  staff  not  familiar  with  automatic  data  proc¬ 
essing.  As  a  result,  about  94  more  staff-months  of  contractor 
effort  were  needed. 

.  The  contractor  stated  that  system  implementation  was 
hindered  by  (1)  agency-requested  system  changes,  (2)  diffi¬ 
culties  in  getting  all  the  bureaus  to  fully  accept  and  use 
the  system,  and  (3)  system  production  problems,  such  as  those 
caused  by  errors  in  both  software  and  keypunching.  At  this 
point  the  agency  approved  an  additional  contractor  proposal 
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to  continue  work  for  a  year  at  a  cost  of  over  $900,000.  That 
proposal  was  supposed  to  insure  more  efficient  operation  and 
to  expand  the  system  to  provide  additional  information. 
Actually,  almost  two-thirds  of  that  cost  was  for  the  contrac¬ 
tor  to  continue  work  he  was  already  doing  rather  than  for 
improvements. 

The  system  was  installed  on  the  computer  at  the  agency's 
data  processing  center.  The  agency  found  that  the  system  was 
too  complex,  that  it  required  too  much  data  preparation,  and 
that  management  could  not  use  its  reports.  The  contractor 
proposed  to  simplify  the  system  and  eliminate  some  reports, 
and  the  agency  approved  additional  work  for  these  purposes. 

An  agency  official  stated  that  both  poor  contractor 
performance  and  agency  failings  contributed  to  the  contrac¬ 
tor's  cost  overrun.  He  stated  that  the  contractor's  project 
manager  did  not  use  professional  system  development  practices 
and  project  control  techniques.  For  example,  programming 
started  before  system  design  was  approved.  The  official 
added  that  the  contractor  did  not  submit  progress  reports 
and  the  agency  did  not  conduct  periodic  technical  reviews. 

At  the  end  of  the  contract  period  the  simplified  ver¬ 
sion  of  the  system  was  incomplete.  The  agency  refused  to 
award  funds  to  continue  work  on  the  system  and  instead,  hired 
two  other  firms  to  make  the  system  operational.  The  new 
contractors  reported  that: 

—Computer  programs  were  in  various  stages  of  comple¬ 
tion  and  ranged  from  barely  adequate  to  useless. 

— Program  testing  was  in  worse  shape  than  reported  and, 
in  their  opinion,  if  any  testing  had  been  done,  no  one 
had  bothered  to  look  at  the  results. 

— Programming  techniques,  documentation,  and  library 
control  all  suffered  serious  deficiencies. 

— To  bring  the  programs  to  a  stage  where  an  integrated 
systems  test  could  be  performed  would  require  a  great 
deal  of  work. 

The  agency  eventually  abandoned  the  use  of  the  original 
system  and  developed  an  in-house  system  for  about  $20,000 
which  basically  performed  the  same  functions  as  the  proposed 
simplified  version  of  the  original  system. 

We  reviewed  the  system  development  documentation  and 
observed  that: 
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— The  agency  assigned  six  different  project  officers 
during  the  life  of  the  contract. 

— The  contractor  assigned  at  least  three  different  de¬ 
velopment  teams  to  the  project. 

—The  contract  was  modified  16  times. 

The  programming  language  that  the  programs  were  to  be 
written  in  was  apparently  not  clearly  specified  ini¬ 
tially.  The  agency  later  instructed  the  contractor 
to  use  COBOL,  but  programs  were  delivered  in  PL-1. 

— Agreements  were  made  between  the  contractor  and  agency 
officials  other  than  the  project  officer. 

Why  this  contract  failed 

The  agency's  assessment  of  its  system  development  sta¬ 
tus  when  it  let  the  contract  was  overoptimistic.  Also,  the 
contract  omitted  important  provisions,  or  mentioned  them 
only  in  general  terms.  For  instance,  satisfactory  perform¬ 
ance  was  not  defined,  the  agency  had  no  recourse  for  nonper¬ 
formance  by  the  contractor,  acceptance  testing  procedures 
were  not  established,  quality  control  procedures  were  not 
required  of  the  contractor,  and  no  documentation  standards 
were  included  in  the  contract.  Thus,  the  agency  could  not 
maintain  control  over  the  type  and  quality  of  the  software 
being  developed. 

The  agency  assumed  that  existing  software  could  be 
adapted  to  the  new  system  and  that  contractor  personnel  who 
were  knowledgeable  of  agency  operations  would  be  available. 
These  assumptions  proved  false.  The  agency's  incorrect 
assumptions — showing  an  incorrect  assessment  of  its  stage  of 
system  development — led  to  two  situations  harmful  to  the 
agency's  interest. 

First,  the  contract  was  awarded  sole-source  without 
competition,  and  second,  the  cost  and  time  estimates  were 
based  on  what  the  agency  thought  had  been  accomplished  and 
the  amount  of  work  thought  to  be  remaining.  If  the  agency 
had  determined  that  it  was  not  technically  feasible  to  adapt 
software  from  the  old  system  to  the  new  system  and  that  the 
^®*^trac tor '  s  supposed  experience  would  thus  be  lost  or  com¬ 
promised,  better  estimates  might  have  shown  that  the  system 
was  not  feasible  to  develop. 

None  of  the  project  officers  the  agency  assigned  to 
work  with  the  contractor  proved  a  strong  focal  point  for  the 
control  and  resolution  of  problems  that  came  up  during  the 
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contract.  To  compound  this  situation,  the  contractor  had  to 
deal  with  seven  bureaus  which,  according  to  the  contractor, 
offered  less  than  their  full  cooperation.  Also,  the  large 
number  of  agency  project  officers  and  contractor  teams 
assigned  to  the  project  made  communications  very  difficult 
and  a  successful  contract  almost  impossible. 

The  agency  lost  another  form  of  control  because  of  the 
lack  of  strict  phasing  of  the  project.  In  this  case,  the 
contractor  began  work  in  the  programming  phase  before  re¬ 
quirements  were  fully  determined;  programming  started  before 
the  system  design  was  approved,  and  the  requirements  and 
users  were  changed  frequently  during  the  contract. 

The  complexity  of  the  delivered  system  and  the  exces¬ 
sive  data  preparation  work  it  required  indicate  that  the 
agency  had  done  little  if  any  technical  review  of  the  con¬ 
tractor's  design  work.  They  also  indicate  a  lack  of  perform¬ 
ance  specifications  and  minimal  system  acceptance  testing. 

The  poor  quality  of  the  product  delivered  by  the  con¬ 
tractor  was  exhibited  by  serious  deficiencies  in  the  program¬ 
ming  technique  and  documentation.  Those  deficiencies  indicate 
that  the  agency  should  have  required  the  contractor  to  demon¬ 
strate  that  quality  control  procedures  were  in  effect  during 
the  software  development  effort.  They  also  indicated  that  the 
agency  should  have  specified  acceptance  testing  requirements. 

In  summary,  the  agency  failed  to  (1)  negotiate  a  contract 
which  identified  and  required  the  specific  level  of  performance 
expected  of  the  contractor,  (2)  recognize  the  complexity  of  the 
system  when  the  extensive  contractor-generated  specifications 
were  presented  to  them,  (3)  determine  what  would  be  needed  to 
support,  use,  and  maintain  such  a  system,  (4)  establish  a 
knowledgeable  project  officer  as  a  single  focal  point  to  deal 
with  the  contractor  and  to  give  that  person  the  necessary 
authority  to  ensure  the  cooperation  of  all  organizations  within 
within  the  agency,  (5)  aggressively  monitor  the  contractor's 
progress  and  pay  for  only  the  agreed  to  levels  of  performance, 
(6)  accomplish  the  system  development  tasks  originally  identi¬ 
fied  as  their  responsibility,  and  (7)  insist  that  the  work  be 
done  in  distinct,  sequential  phases  with  adequate  technical 
review.  The  result  was  a  large,  complex  system  which  cost 
many  times  the  original  estimate  and  was  useless. 

Without  a  doubt  the  contractor's  work  was  inadequate. 
However,  the  agency  should  have  written  and  managed  the  con¬ 
tract  to  ensure  that  it  had  some  recourse  if  the  delivered 
product  proved  to  be  useless. 
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Case  Study  3  -  The  payroll/personnel  system  that  wasted 
$1  million 

The  agency  let  a  contract  to  design  an  integrated 
personnel/payroll  system,  redesign  an  expense  accounting  sys¬ 
tem,  prepare  an  equipment  requirements  statement,  and  develop 
a  background  study  on  all  its  ADP  systems.  The  original  con¬ 
tract  cost  and  time  were  $445,158  and  15  months,  most  of  which 
was  for  the  software  development  portion  of  the  contract. 

The  agency  finally  terminated  the  contract  after  28  months  and 
$970,000  and  with  no  usable  software  being  delivered. 

What  happened  during  the  contract 

When  it  issued  a  request  for  proposal,  the  agency  had 
only  identified  a  need  for  the  software  and  was  still  in 
the  initial  stages  of  system  development.  It  had  not  fully 
developed  user  requirements  or  system  specifications  for 
any  of  the  proposed  software. 

The  agency  selected  a  contractor  and  awarded  a  fixed- 
price  contract.  The  contract  required  that  the  software  de¬ 
velopment  be  phased  but  did  not  require  agency  approval  of 
a  completed  phase  before  work  began  on  the  next  phase.  Pro¬ 
cedures  were  written  to  control  changes  in  the  scope  of  the 
work,  and  the  programming  language  was  specified.  The  con¬ 
tract  only  described  satisfactory  performance  by  the  con¬ 
tractor  in  such  general  terms  as  "the  system  must  demonstrate 
effectiveness,  be  automated  wherever  possible,  and  be  flex¬ 
ible.^  The  contract  did  not  contain  acceptance  testing  pro¬ 
cedures  and  the  documentation  the  contractor  was  to  produce 
was  identified  in  the  contract,  but  the  quality  criteria  for 
it  were  not. 

The  contract's  delivery  dates,  scope  of  work,  and  costs 
were  revised  several  times,  yet  the  contractor  was  still  un¬ 
able  to  meet  the  revised  schedule  or  deliver  an  acceptable 
product  to  the  agency.  The  contractor  stated  that  he  was  un¬ 
able  to  meet  the  delivery  schedule  due  to  extensive  changes 
requested  by  the  agency  and  inexcusable  delays  caused  by  the 
agency.  Agency  officials  admitted  that  some  of  the  changes 
requested  were  not  clearly  identified  in  the  contract  and 
that  others  were  clearly  outside  the  scope  of  work. 

The  contractor  maintained  that  the  agency  took  too  much 
time  to  review  products  submitted  for  approval.  The  agency 
admitted  that  the  reviews  were  not  timely  but  maintained 
that  the  length  of  the  review  was  due  to  the  poor  quality  of 
the  documentation  to  be  reviewed. 
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The  contractor  did  not  clearly  understand  the  software 
systems  the  agency  desired  because  the  contract  did  not  spe¬ 
cify  system  requirements  or  performance  criteria.  Both 
agency  and  contractor  staff  agreed  that  the  contract  was  not 
specific,  that  the  terminology  was  vague,  and  that  many  sys¬ 
tem  requirements  were  not  clearly  identified. 

The  contract  included  a  milestone  chart  showing  start¬ 
ing  and  completion  dates  for  phases  of  system  development. 

The  contractor  requested  the  agency's  approval  after  phases 
were  completed  but  did  not  wait  for  that  approval  before 
starting  work  on  succeeding  phases.  For  example,  the  con¬ 
tractor  began  work  on  the  detailed  system  design  before  re¬ 
ceiving  agency  approval  of  the  general  system  design.  Later, 
the  agency  rejected  the  general  design  and  required  the  con¬ 
tractor  to  make  several  changes,  so  work  already  done  on  the 
detailed  design  had  to  be  scrapped  or  reworked. 

User  requirements  were  never  adequately  defined  and 
frozen.  The  agency  provided  system  concepts  for  two  of  the 
systems,  and  the  contractor  worked  with  current  system  users, 
reviewed  available  system  documentation,  and  developed  user 
requirements  based  on  responses  to  a  questionnaire.  The  con¬ 
tractor  used  this  information  to  develop  the  general  design 
which  the  agency  rejected  several  times  for  not  meeting  re¬ 
quirements.  Each  time  the  agency  rejected  the  design,  it 
added  new  requirements.  The  changes  delayed  completion  sched¬ 
ules,  increased  contract  costs,  and  caused  the  agency  and  the 
contractor  to  disagree  about  whether  the  new  requirements 
were  included  in  the  original  scope  of  work. 

The  contract  was  amended  13  times  to  provide  for  addi¬ 
tional  work  to  be  done,  to  add  or  delete  requirements,  and 
to  reimburse  the  contractor  for  extra  costs  due  to  delays 
caused  by  the  agency.  The  amendments  were  to  increase  the 
cost  of  the  contract  to  $1,037,448.  The  agency  eventually 
became  convinced  that  the  contractor  could  not  deliver  at  an 
acceptable  time  and  cost,  cancelled  the  contract,  and  tried 
to  withhold  payment  for  poor  performance.  A  negotiated  set¬ 
tlement  price  of  $970,000  was  finally  agreed  to.  None  of 
the  software  was  ever  used  by  the  agency. 

Why  this  contract  failed 

The  fixed-price  contract  was  not  suitable  since  the 
agency  could  not  provide  detailed  specifications  for  the  soft¬ 
ware  desired.  Because  the  contract  did  not  include  detailed 
specifications,  it  had  to  be  modified  13  times  to  accommodate 
changes,  extend  delivery  dates,  and  provide  more  money.  One 
form  of  cost-plus  contract  is  normally  better  for  cases  where 
the  exact  scope  of  work  has  not  been  identified.  In  this  case. 
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however,  the  contractor  wanted  to  limit  his  efforts  to  make 
a  profit  at  the  fixed  price  he  bid.  The  relationship  between 
the  agency  and  the  contractor  deteriorated  as  requirements 
were  added,  delays  incurred,  and  the  contractor  saw  his  pro¬ 
fit  shrink. 

Phasing  was  not  used  to  good  advantage  in  this  contract. 
The  system  development  was  not  broken  into  fixed  phases  with 
agency  approval  required  for  each  before  proceeding.  As  it 
was,  even  the  requirements  were  still  under  dispute  when  the 
contract  ended.  The  parties  could  not  use  the  contract  to 
settle  disputes  because  it  was  written  in  such  general  terms. 

The  areas  in  which  those  generalities  caused  significant 
disputes,  delays,  and  added  costs  were  acceptance  testing, 
performance  specifications,  and  documentation. 

In  summary,  the  agency  wrote  a  contract  which  was  (1) 
the  wrong  type  for  the  stage  of  system  development  they  were 
in  when  it  was  let  and  (2)  too  general  to  give  the  contrac¬ 
tor  a  clear  understanding  of  what  was  to  be  done. 

Also,  the  agency  did  not  use  phasing  to  control  the 
development  process  and  managed  the  contract  so  poorly  that 
the  contractor  could  show  that  the  agency  contributed  to  the 
failure  to  produce  usable  software  and  was  thus  able  to  de¬ 
feat  the  agency's  attempt  to  withhold  payment. 
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Case  Study  4  -  The  failure  to  develop  a  centralized 
accounting  system 

The  agency,  which  used  nonuniform  accounting  systems, 
contracted  for  the  design  and  development  of  a  centralized 
accounting  system  to  increase  responsiveness,  timeliness, 
accuracy,  and  to  overcome  inefficiencies  in  the  operation  of 
10  aiccounting  offices  within  the  agency*  The  cost  and  time 
spent  were  estimated  to  be  $958,682.40  and  27  months.  After 
30  months  the  system  was  only  about  one-fourth  complete,  and 
the  agency  cancelled  the  contract.  Although  the  system  was 
not  complete,  the  agency  paid  about  $981,200. 

What  happened  during  the  contract 

At  the  time  the  contract  was  let,  the  agency  had  no 
formal  design  or  specification  documents  for  the  contractor 
to  work  from.  The  agency  had  collected  a  list  of  concepts 
and  standards  which  supposedly  were  the  basis  of  a  conceptual 
design.  The  cost-plus-fixed-fee  contract  obligated  the  con¬ 
tractor  to  deliver  a  workable  accounting  system. 

The  contract  called  for  three  development  phases.  Each 
covered  the  development  of  a  major  accounting  subsystem,  with 
the  first  phase  to  include  the  overall  system  design.  Each 
phase  was  further  divided  into  conceptual  design,  detail 
design,  and  implementation.  The  agency  was  to  approve  each 
phase  at  completion.  Under  the  terms  of  the  contract,  the 
contractor  was  to  develop  a  project  control  plan  for  such 
items  as  progress  and  cost  reporting,  documentation  review, 
and  acceptance  testing.  The  contractor  was  responsible  for 
formulating  the  criteria  by  which  the  agency  would  judge  his 
performance.  The  contract  called  for  the  contractor  to  sub¬ 
mit  proposed  changes  along  with  the  reasons  for  them  to  the 
agency  for  approval.  The  agency  reserved  the  right  to  make 
modifications  it  considered  necessary  to  ensure  that  the 
system  fit  its  needs.  System  documentation  requirements 
were  fairly  detailed,  but  guidance  on  program  documentation 
only  referred  the  contractor  to  agency  standards.  No  sub¬ 
contractors  were  involved. 

In  the  first  phase,  the  contractor  was  to  develop  a 
general  design  of  the  overall  system  and  also  the  design  for 
one  major  subsystem.  In  the  development  of  this  phase  the 
contractor  stated  that  he  encountered  two  problems — (1)  the 
agency  staff  generally  resisted  the  new  system  and  (2)  virtu¬ 
ally  none  of  the  existing  accounting  processes  and  procedures 
which  the  new  system  was  to  automate,  was  documented.  When 
he  submitted  his  report  on  the  first  phase,  the  contractor 
assumed  he  could  immediately  start  on  the  next  phase,  but 
agency  review  of  the  report  took  about  250  staff— days.  The 
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contractor  said  that  agency  delays  and  the  low  level  of  agency 
participation  together  added  about  350  staff-days. 

As  the  contractor  entered  the  second  part  of  the  first 
phase,  he  still  encountered  problems  he  blamed  on  (1)  the 
poor  quality  of  agency  review  and  agency  staff  participation, 
(2)  agency  indecision,  and  (3)  agency  changes  in  direction. 

The  contractor  felt  the  changes  deviated  from  earlier  agree¬ 
ments  and  that  some  of  them  could  not  be  made.  Some  pro¬ 
ducts  were  submitted  for  agency  approval  three  or  four  times. 
Disagreements  arose  over  the  amount  of  documentation  neces¬ 
sary,  and  the  lack  of  existing  agency  procedures  continued 
to  be  a  problem. 

The  contractor  contended  that  the  agency  insisted  on  a 
system  that  was  not  needs-oriented  but  one  which  was  designed 
to  satisfy  many  individual  preferences.  To  illustrate  his 
point,  he  compared  the  excessive  number  of  management  reports 
asked  for  in  this  system  (188)  to  the  maximum  number  of  re¬ 
ports  (44)  called  for  in  four  other  agencies'  accounting  sys¬ 
tems.  The  agency  director  of  systems  admitted  the  general 
lack  of  direction  and  specifics  in  the  contract  and  stated 
that  more  definitive  planning  and  guidance  were  needed  to 
let  the  contractor  know  what  was  expected. 

To  determine  what,  where,  and  how  data  would  be  stored 
and  retrieved,  the  contractor  asked  the  agency  to  specify 
(1)  the  computer  hardware  to  be  used,  (2)  the  data  base  man¬ 
agement  system  (DBMS)  1/  to  be  used,  and  (3)  system  require¬ 
ments,  such  as  output  reports  and  transaction  coding.  The 
agency  took  about  6  months  to  provide  guidance  in  these  areas, 
and  during  that  time,  the  contractor  proceeded  to  design  con¬ 
ventional  file  structures  and  processing  routines. 

When  the  agency  finally  decided  on  the  DBMS,  the  design 
had  to  be  substantially  reworked.  The  contractor  said  that 
even  small  changes  generated  extensive  reviews  to  determine 
all  other  areas  which  were  affected  and  required  change.  Of 
the  six  major  reasons  given  by  the  contractor  for  overruns, 
three  dealt  with  changes  that  were  constantly  being  made  to 
both  the  requirements  and  the  operational  environment  and 
the  impact  those  changes  had  on  system  development. 


1/A  Data  Base  Management  System  (DBMS)  is  a  computer  software 
package  which  can  facilitate  the  management,  manipulation, 
and  control  of  data. 
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Agency  officials  maintained  that  the  contractor's  report 
reflected  a  general  lack  of  understanding  of  the  job  to  be 
done,  that  deliverables  were  inadequate,  and  that  agency 
documentation  standards  were  not  followed.  Conversely,  the 
contractor  stated  that  agency  staff  assigned  to  work  on  the 
subject  did  not  understand  the  agency,  its  mission,  or  its 
needs. 

After  about  2-1/2  years,  the  agency  had  paid  the  con¬ 
tractor  over  $981,000.  The  system  was  estimated  to  be  only 
about  one-fourth  complete,  and  the  time  frame  had  exceeded 
the  original  estimate  by  several  months.  At  this  point  the 
agency  terminated  the  contract.  The  contracting  officer 
said  that  the  agency's  counsel  had  informally  advised  him 
that  a  precisely  defined  set  of  requirements  was  never  in¬ 
corporated  in  the  contract.  Therefore,  since  neither  party 
could  define  the  product,  in  their  unofficial  opinion,  the 
agency  could  not  force  the  contractor  to  finish  the  system 
for  the  maximum  cost  allowed  by  the  contract. 

Why  this  contract  failed 

Too  many  factors  were  left  to  be  subjectively  deter¬ 
mined  outside  the  provisions  of  the  contract.  This  condi¬ 
tion  is  evidenced  by  such  things  as  arbitrary  changes,  dis¬ 
agreements  on  various  subjects,  and  the  agency's  admissions 
that  more  specific  requirements  should  have  been  given  to 
the  contractor.  These  problems  can  be  avoided  even  if  the 
exact  characteristics  of  the  needed  software  are  not  known 
at  the  time  the  contract  is  let. 

First,  user  requirements,  performance  specifications, 
quality  control  procedures,  acceptance  testing  procedures, 
and  documentation  items  required  can  be  specified  in  the  con¬ 
tract  to  establish  a  framework  at  the  outset.  Second,  the 
agency  should  have  required  the  overall  system  design  to  be  _ 
defined  in  a  first  phase  to  the  point  that  its  adequacy  could 
have  been  determined,  approved,  and  frozen  to  allow  stable 
and  systematic  development  before  committing  itself  to  the 
rest  of  the  contract. 

Other  factors  which  contributed  to  the  failure  of  the 
contract  included  the  agency's  failure  to; 

— Establish  firm,  realistic  requirements  and  fix  them. 

—Render  timely  decisions  and  timely  review  of  products. 

— Promptly  carry  out  responsibilities  so  that  software 
development  was  not  delayed  and  so  that  the  way  was 
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left  clear  to  invoke  contract  penalties  against  the 
contractor  if  he  failed  to  perform. 

— Maintain  adequate  monitoring  and  tracking  procedures 
which  would  have  avoided  allowing  the  entire  original 
contract  amount  to  be  spent  in  the  first  of  three  de¬ 
velopment  phases.  ' 

— Define  the  user  requirements  served  by  the  existing 
accounting  system. 

— Create  an  environment  which  enhanced  chances  for  suc¬ 
cess,  including  consensus  on  agency  needs,,  proper 
orientation  of  agency  4!t:aff  ,y::;^^  provision  of  a  strong 
focal  point  of  qualified  agency  staff  to  work  with 
the  contractor. 
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Case  Study  5  -  The  successful  compiler  development 

In  this  case  the  agency  contracted  for  the  development, 
implementation,  and  documentation  of  a  SIMSCRIPT  II  language 
compiler  for  the  agency's  computers.  The  contract  called 
for  the  compiler  to  be  completed  and  delivered  in  1  year  at 
a  cost  of  $119,800.  The  software  was  delivered  within  the 
cost  and  within  a  month  of  the  delivery  date  with  final 
testing  to  be  accomplished  shortly  afterward. 

What  happened  during  the  contract 

The  agency  had  developed  detailed  specifications.  Also, 
the  agency  had  developed  proposal  evaluation  criteria,  which 
included  acceptance  testing  criteria.  The  fixed-price  con¬ 
tract  specified  the  system  requirements  as  well  as  what  was 
expected  of  the  contractor.  It  included  a  26-page  statement 
of  work  with  several  appendexes  detailing  the  required  and 
desirable  system  specifications. 

The  agency's  acceptance  of  the  product  was  to  be  based 
on  the  software's  successful  completion  of  a  final  comprehen¬ 
sive  benchmark;  however,  interim  tests  were  to  be  conducted 
throughout  the  performance  period.  The  contractor  was  obli¬ 
gated  to  correct  any  deficiencies  found  in  the  product  for 
up  to  3  months  after  delivery  at  no  additional  cost.  The 
contract  cited  the  exact  documentation  expected  and  also  pro¬ 
vided  for  the  contractor  to  train  agency  personnel  to  use  the 
compiler.  Contract  provisions  required  that  the  contractor's 
principal  staff  on  the  job  have  experience  in  developing  and 
implementing  large-scale  simulation  language  compilers,  and 
that  they  have  at  least  2  year's  extensive  use  of  the  SIM¬ 
SCRIPT  language.  The  contractor's  progress  was  to  be  moni¬ 
tored  by  on-site  reviews  at  the  contractor's  facilities. 

To  begin  with,  the  agency  developed  good  proposal 
evaluation  criteria.  The  project  officer  stated  that  the 
criteria  were  developed  to  provide  a  way  to  reject  vendors 
who  would  probably  be  unable  to  do  the  job.  Second,  he 
wanted  to  evaluate  the  proposals  in  terms  of  their  dollar 
benefits,  considering  the  future  savings  to  be  generated  by 
each  rather  than  just  their  development  cost.  Finally,  he 
wanted  to  be  able  to  differentiate  between  the  qualified 
vendors.  This  project  officer  was  deeply  involved  through¬ 
out  the  proposal  evaluation  and  prepared  the  majority  of  the 
specifications  and  requirements. 

To  accomplish  these  aims,  the  project  officer  devised  a 
41-step  weighted  rating  system  to  rate  the  proposals  in 
three  areas:  the  vendor's  compliance  with  mandatory  contract 
requirements;  the  vendor's  capabilities,  qualifications,  and 
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experience;  and  the  vendor's  predicted  product  performance 
and  dollar  benefits  to  the  agency.  He  also  inserted  several 
clauses  in  the  contract  that  he  knew  were  technically  infea¬ 
sible,  to  trap  ignorant  vendors. 

The  contract  also  contained  a  clause  allowing  agency 
representatives  to  visit  the  vendor's  facility  to  determine 
his  ability  to  perform.  This  visit,  if  conducted,  could  in¬ 
vestigate  and  evaluate  current  financial  position;  technical, 
production,  and  financial  capability;  plant  facilities  and 
equipment;  quality  assurance;  labor  resources;  performance 
record;  and  other  appropriate  subjects. 

The  project  officer  was  also  concerned  with  the  quality 
of  the  functional  system  specifications.  The  procurement 
officer  said  they  were  the  best  he  had  seen  for  a  software 
procurement;  the  clear  specifications  allowed  him  to  offer 
the  contract  on  a  competitive  basis.  The  contractor  stated 
the  specifications  made  it  very  clear  what  the  agency  wanted. 

The  software  development  incorporated  benchmark  tests 
which  were  conducted  throughout  the  development  cycle.  A 
benchmark  monitor  was  designated  by  the  agency  to  develop 
the  benchmark  programs,  conduct  the  tests,  and  prepare  the 
test  results.  The  agency  specified  the  exact  documentation 
the  contractor  was  to  provide,  including  user  manuals,  sys¬ 
tems  programmers'  manuals,  complete  flow  charts,  and  com¬ 
plete  source  code  to  maintain  the  compiler  after  the  contract 
expired  and  to  facilitate  competitive  contract  maintenance. 
Training  was  provided  by  the  contractor  to  allow  agency  per- 
personnel  to  conduct  followon  systems  maintenance.  The  con¬ 
tractor  delivered  all  documentation  and  training  to  the 
agency's  satisfaction. 

With  his  experience  in  ADP  in  general,  and  in  simulation 
languages  in  particular,  the  project  officer  was  equipped  to 
monitor  the  contractor's  progress.  To  track  and  evaluate 
the  contractor,  he  incorporated  the  benchmark  testing  process 
into  the  contract  and  conducted  periodic  on-site  reviews  with 
contractor  and  agency  staff. 

Each  on-site  progress  review  lasted  2  or  3  days.  Accord¬ 
ing  to  the  project  officer's  trip  reports,  at  these  meetings 
he  was  apparently  able  to  determine  how  well  the  contractor 
was  progressing,  to  discuss  and  resolve  problems,  and  to  gen¬ 
erally  assure  himself  that  the  contractor  was  adequately  ful¬ 
filling  his  contractual  obligations.  Further  assessment  of 
contractor  progress  was  provided  by  the  periodic  benchmark 
tests.  Also,  throughout  the  project's  development,  contractor 
and  agency  personnel  communicated  openly  and  frequently  to  as¬ 
sess  contractor  progress  and  to  resolve  questions  that  arose. 
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Delivery  of  the  software  was  delayed  1  month  because 
the  contractor  could  not  obtain  documentation  from  the  hard¬ 
ware  vendor  for  the  execution  library.  The  agency  considered 
this  delay  unavoidable  and  it  was  mutually  agreed  upon.  After 
some  minor  problems  were  corrected,  the  software  passed  the 
final  benchmark  test  on  September  15,  1972,  and  final  payment 
was  made  to  the  contractor. 

According  to  the  agency,  it  received  an  excellent  pro¬ 
duct.  The  compiler  was  usable  immediately  after  acceptance 
and  saved  the  agency  an  estimated  $65,000  annually  because 
of  its  increased  efficiency.  Also,  the  contractor  delivered 
the  documentation  according  to  plan. 

Why  this  contract  succeeded 

This  case  is  an  example  of  successful  software  develop¬ 
ment  contracting.  Several  very  important  factors  were  in 
its  favor: 

— -The  agency  had  an  extremely  diligent  and  thorough  proj¬ 
ect  officer  who  was  extremely  competent  and  techni¬ 
cally  knowledgeable.  This  was  verified  by  a  contrac¬ 
tor  official  and  the  agency's  contracting  officer, 
who  said  that  the  project  officer's  functional  speci¬ 
fications  made  it  possible  to  have  a  competitive 
procurement.  Moreover,  this  project  officer  stayed 
with  the  project  from  start  to  finish. 

— An  extensive  contractor  rating  and  proposal  evaluation 
scheme  was  developed  which  ensured  that  only  capable 
contractors  would  be  selected. 

— Excellent  functional  specifications  were  developed 
including  extensive  benchmark  tests  (both  during  de¬ 
velopment  and  during  acceptance) ,  documentation,  and 
training . 

— Periodic  reviews  and  open  communication  were 
maintained  throughout  the  contract. 

— A  type  of  computer  programming  task  was  included  which 
could  definitely  be  made  a  fixed-contract  requirement. 
That  is,  the  compiler  was  to  translate  a  given  program¬ 
ming  language  (SIMSCRIPT  II)  into  machine  language  for 
a  specific  brand  of  computer.  The  language  did  not 
change  during  the  contract.  Also,  the  performance  and 
efficiency  of  the  compiler  could  be  more  easily  quanti¬ 
fied  to  provide  benchmark  test  criteria.  The  relative 
importance  of  having  a  programming  task  less  changeable 
than  the  typical  business  programming  task  is  difficult 
to  assess.  However,  we  believe  that  it  is  important. 
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Case  Study  6  -  The  contract  test  scoring  program  that 
failed  its  own  test  and  had  to  be 
rewritten  by  agency  programmers 

The  agency  contracted  for  the  development  and  implemen¬ 
tation  of  a  data  storage,  retrieval,  and  editing  system  to 
be  used  in  connection  with  a  certification  test  program  the 
agency  conducts.  The  tasks  to  be  accomplished  consisted  of 
three  major  data  processing  programs,  a  test  data  sheet,  and 
19  additional  auxiliary  computer  programs.  The  programs  were 
to  make  the  testing  easier  and  to  automate  much  of  the  later 
analysis  and  retrieval  of  test  results.  No  detailed  descrip¬ 
tion  of  these  programs  was  included  in  the  contract.  Origi¬ 
nally  the  contract  was  supposed  to  be  completed  in  6  months 
at  a  cost  of  about  $14,000.  The  contract  lasted  about 
9  months  without  usable  software  being  delivered.  Even  though 
no  usable  software  was  delivered,  the  agency  still  paid  the 
contractor  about  $13,000. 

What  happened  during  the  contract 

Agency  officials  stated  that  when  the  contract  was  let, 
they  had  identified  "fairly  specific"  system  requirements  and 
had  developed  a  general  system  design.  They  wanted  their 
existing  program  to  be  drastically  modified  to  make  it  more 
efficient  and  to  provide  more  data.  The  agency  apparently 
perceived  itself  to  be  well  into  the  system  development  cycle 
when  it  let  the  contract. 

The  contract  was  written  generally  without  detailed  task 
descriptions.  The  agency  relied  on  verbal  discussions  with 
the  contractor  to  define  requirements.  Provisions  were  made 
in  the  contract  for: 

—All  code  to  be  written  in  Fortran  IV,  internally  doc¬ 
umented  (commented)  V  enough  that  an  equally  skilled 
programmer  could  easily  modify  and  enhance  the  pro¬ 
grams. 


The  contractor  to  provide,  whenever  possible,  input/ 
data  base  file  formats  and  layouts,  outputs  layouts, 
and  system  flow  diagrams  on  interrelationships  between 
data  files,  data  flow,  program  utilization,  and  file 
usage . 


1/  Comments  in  FORTRAN  are  notes  for  persons  to  read  that 
are  embedded  among  the  actual  FORTRAN  computer  language 
statements  of  the  program. 
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The  contract  did  not  provide  any  procedures  by  which 
the  agency  could  monitor  the  contractor's  performance.  In 
fact/  it  stated  that  written  progress  reports  from  the  con¬ 
tractor  would  not  be  necessary.  The  contract  did  not  estab¬ 
lish  any  procedures  which  would  have  facilitated  monitoring, 
such  as  phasing,  defining  specific  benchmarks  and  acceptance 
testing  procedures,  establishing  procedures  to  control  changes 
in  the  work  scope,  and  specifying  detailed  documentation 
standards. 

The  contract  does  not  clearly  describe  what  work  was 
to  be  performed  or  what  products  were  to  be  delivered.  Agency 
officials  later  stated  that  they  had  a  verbal  agreement  with 
the  contractor  and  that  both  parties  clearly  understood  the 
anticipated  system  design,  the  system  specifications,  the 
user  requirements,  and  the  software  application.  Further, 
agency  officials  said  it  was  clear  that  they  expected  the 
contractor  to  deliver  finalized  programs  in  time  for  imple¬ 
mentation  at  a  specified  point.  The  agency  said  the  contrac¬ 
tor  was  involved  in  early  discussions  of  the  task  and  had 
agreed  with  the  agency's  estimated  number  of  hours  required 
to  complete  the  work.  The  contractor  stated  that  the  agency 
had  not  developed  details  of  the  task  and  that  he  was  engaged 
in  part  to  help  formulate  these  details. 

Although  only  two  official  modifications  were  made  to 
the  contract,  we  found  evidence  that  several  informal  changes 
were  made  during  the  performance  period.  The  two  formal  mod¬ 
ifications  did  not  affect  the  substance  of  the  work  to  be 
performed;  they  designated  a  new  agency  project  officer  and 
extended  the  contract  completion  date. 

The  informal  changes  were  more  substantial.  The  agency 
requested  several  changes  to  input  and  output  data  formats 
within  the  next  several  months  after  the  contract  was  let. 

The  agency  and  the  contractor  disagreed  on  their  significance. 
Agency  officials  believed  that  the  changes  were  not  substan¬ 
tial  and  cited  the  fact  that  the  contractor  never  indicated 
that  the  changes  were  causing  him  delays  or  inconvenience. 

The  contractor,  however,  stated  that  the  changes  did  cause 
delays. 

Agency  officials  underestimated  the  complexity  and  the 
amount  of  programming  support  needed  to  complete  one  pro¬ 
gram.  They  estimated  192  hours,  but  contractor  staff  actu¬ 
ally  spent  more  than  350  hours  on  it  and  estimated  that  220 
more  hours  would  be  required. 

The  agency  relied  on  informal  monitoring  and  did  not 
require  formal  progress  reports  from  the  contractor.  The 
agency  project  officer,  who  was  the  contractor's  primary 
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contact  with  the  agency,  stated  that  he  relied  on  the 
contractor's  verbal  reports  to  monitor  progress.  Later  he 
said  that  he  felt  the  contractor  staff  wasted  time  when  they 
were  not  watched. 

The  agency  did  not  receive  usable,  completed  software. 
When  the  contractor  could  not  successfully  demonstrate  the 
product  about  a  month  before  the  scheduled  completion,  the 
agency  realized  it  would  not  be  finished  by  the  end  of  the 
contract  period.  None  of  the  computer  programs  finally  de¬ 
livered  was  suitable.  The  first  program  was  inefficient, 
costing  between  $250  and  $300  per  day  to  run  and  over  $2  to 
generate  a  single  test  score.  It  contained  many  programming 
errors  and  was  not  adequately  documented.  The  agency  never 
even  tried  to  use  the  second  program  the  contractor  delivered. 
Agency  programmers  eventually  wrote  a  program  which  cost 
about  90  percent  less  to  run  than  the  first  program  deliv¬ 
ered.  The  agency  withheld  a  final  payment  of  $1,351.81  from 
the  contractor  because  of  the  inadequacy  of  the  software 
delivered. 

Why  this  contract  failed 

The  agency  failed  to  correctly  assess  the  system  devel¬ 
opment  progress  it  had  made  before  letting  the  contract. 

While  agency  officials  stated  that  they  had  identified  re¬ 
quirements  and  completed  the  general  system  design  stage, 
both  the  generalities  of  the  contract  documents  and  the  con¬ 
tractor's  testimony  made  it  clear  that  this  was  not  true. 

Since  the  agency  had  not  completed  a  system  design  and  did 
not  ask  the  contractor  to,  development  work  was  begun  without 
a  foundation  of  design.  Also,  without  a  design,  criteria  for 
acceptance  testing  could  not  be  developed  even  if  the  agency 
had  attempted  to  include  it  in  the  contract. 

The  vaguely  written  contract  did  not  make  the  contrac¬ 
tor's  responsibilities  clear.  Verbal  discussion  was  substi¬ 
tuted  for  formal  contract  requirements,  and  the  lack  of 
communication  between  the  agency  and  the  contractor  never 
allowed  the  scope  of  work  to  be  clarified. 

The  agency  made  no  provision  for  monitoring  the  contrac¬ 
tor's  progress  or  for  reviewing  and  approving  work  phases. 
Consequently,  the  agency  was  not  aware  of  problems  until  it 
was  nearly  time  for  the  product  to  be  delivered.  Further¬ 
more,  the  agency  defined  the  work  to  be  done  so  vaguely  that 
little  basis  existed  on  which  interim  performance  could  be 
evaluated.  No  formal  progress  reports  from  the  contractor 
were  required,  and  changes  in  the  scope  of  work  were  handled 
informally  by  the  agency.  According  to  the  contractor,  these 
changes  had  a  great  impact  on  the  project  and  caused  the  ori¬ 
ginal  completion  date  to  be  expanded. 
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In  sununary^  the  agency  wrote  a  contract  which  (1)  did 
not  clearly  define  and  document  what  was  expected  from  the 
contractors,  (2)  did  not  phase  development  with  review  and 
approval  of  each  phase,  (3)  did  not  provide  for  the  contrac¬ 
tor's  progress  to  be  adequately  monitored  during  the  course 
of  the  project,  and  (4)  was  based  on  a  stage  of  system  devel- 
OExnent  not  yet  reached. 
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Study  7  -  The  three-party  contract  that  failed  to 
produce  a  usable  accounting  system 

.  . this  case,  a  Federal  agency  contracted  the  adminis¬ 

tration  of  one  of  its  functions  to  an  association.  The 
association  is  responsible  for  maintaining  accounting  and 
administrative  records  of  the  function *s  operations.  The 
association  then  contracted  with  a  computer  firm  to  develop 
and  implement  a  computerized  accounting  system.  For  purposes 
of  this  analysis,  the  three  parties  will  be  referred  to  as 
the  agency,  the  association,  and  the  contractor. 

t-hc  accounting  system  designed  and  implemented  by 

the  contractor,  the  association  was  unable  to  prepare  finan¬ 
cial  statements  and  to  properly  identify  the  function's  oper- 

reviewed  the  association's  records 
and  concluded  that  if  an  audit  could  be  performed  at  all,  it 

so  extensive  that  the  cost  would  make  it 
prohibitive.  Considering  system  and  contractual  problems  to 
be  insurmountable,  the  association  cancelled  the  contract 

control  of  the  system.  At  this  point  the  asso- 
ciation  had  paid  about  $4.3  million  of  the  $5.7  million 
billed  by  the  contractor. 

What  happened  during  the  contract 


The  association  used  system  specifications  prepared  under 
a  previous  contract  to  solicit  bids  for  the  developLnt  of  the 
accounting  system.  Even  though  the  specifications  were  used 
ho  for  the  solicitation,  they  were  later  estimated  to 

be  only  from  50  percent  to  80  percent  complete. 

total  contract  cost  an  estimated  $16  million  over 
^  1  purposes  of  this  study  we  used  only 

inipleraentation,  and  conversion  costs.  Those 
costs  were  to  total  $1,308,252  with  work  exceeding  the  scope 

be  quoted  on  the  basis  of  time^nd  mate?! 
lals  used  based  on  rates  listed  in  the  contract.  The  con- 
tract  called  for  the  work  to  be  done  in  four  stages  but  did 

5*?®^  ®®®^  stage  would  be  completed  and  approved 
dLd^  before  work  could  begin  on  the  next?^  In¬ 

deed,  the  milestone  chart  indicated  that  work  might  be  done 
concurrently  on  all  stages.  ^  ® 

The  contract  required  the  contractor  to  draft  an  acceot- 

sti?ulItL^ih?i-^?h  ®^®^®  development.  The  contract 

stipulated  that  the  agency  could  withhold  payment  for  each 

calendar  day  s  delay.  The  contractor  was  also  liable  for  dam- 
ages  sustained  by  the  association  because  of  negligence  or 
failure  to  perform  unless  it  was  caused  by  the  Isso?U?i??. 
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be£ore1h“%"o«ractorc“uS'd*™Z‘I;%"?=°‘^'^“°'’ 

roing  language  was  specif ieH  in  i-h  The  program- 

?h;  dau  P^I«ss‘iSrlndL?r'r"'and  J---^P°b^a™able“iS 

The  contract  was  let  in  Auoust  1974  ttt 

pleted  in  Mav  1975  tho  aeo^,,*  *.•  Stage  III  was  com- 

the  system'^development^^"ln^thP°"™^^^^^  approved  Stage  I  of 
that  Stage  I  progranminq  appeared  letter  they  stated 

portions  of  stage  i:  programming  were  Lrladrweil  uSwayf 

problems*’were*^enoount«ed*i™ldlately  af tlr°the’^^®®t 

Placed  in  operation  and  thr™|hout  Wlsf  . 

lems  incurred  makes  it  appear  that  the  miaTv^  P^°^- 

ciations'  review  of  the  in  n  quality  of  the  asso- 

ciation  later  concluded  that  poor.  The  asso- 

were  grossly  incorrect.  ^  ^  generated  by  the  system 

The  problems  became  so  severe  that  eveni-nai  i..  ..u 
ciation  was  unable  to  nrer>a»-^a  ei  ^.^5  ®'^entually  the  asso- 

results  of  its  operatio^q^  mhf  statements  on  the 

tributed  this  inability  to  multiole^f ^^^^^^oUer  at- 
problems  caused  by  the  failure  statistical 

meet  his  oontractLl  IbUgitllnl  5L°as?;;^®^  • 
a  CPA  firm  to  review  the  oroM^^o  association  requested 

tract.  The  firr^lportL^^^^i  ?k  with  the  con- 

the  criteria  established  by  the^associatio^  "‘eet 

of  program  operations  and  cited  about  75  accountability 

22  major  system  requirements.  deficiencies  covering 

tracto?  tfpay'aboit'^O  pf^cenf o?" an  offer  to  the  con- 
had  been  withheld  to  settle  the  contrLt^^^^h  which 

manded  formal  arbitration.  On  N^vembl?  1 

tion  proceedings  were  termin«t«H  k,,  I  1976,  the  arbitra- 

parties.  The  LrminatJ^n  ag?elLn?  was“?orma??"""  """ 
containing  a  settlement  of  $4?1  million  executed 

claim  of  $5.7  million  contractor's 

the  CPA  firm  over  $1  million  P^^^ 

provide  support  for  arbitMtioI!  review  the  system, 

planning.  The  association  aee”'  conduct  system 

the  contract  was  terminated  control  of  the  system  once 
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‘"estlb? the  system  have  had  to  be 
essed!  ^  ^  ^  assure  the  integrity  of  the  data  proc- 

"daL!^^^^""  provide  the  needed  statistical 

T^;^;:.“rJST:3‘S;.!;;ri.rgr„-:sn,. 

tency  in  program  terminology), 

--The  system  is  not  designed  to  facilitate  modifica¬ 
tions  to  meet  changing  requirements. 

struction  and  operation  of  the  system  ° 

the^data  provided  by  the  association  «  systerinpuriontaL^ed 

requireLn?rKd^orKe^°inSudJI\r?he“lnft-'’?‘ 

vided  the  technical  expertise  needed't^'ad  agency  pro- 

the  contractor's  development  of  the  sJOtOm!  ^  ^  monitor 

to  suOMssful'^TOStrOctiM''°Ond'^J®*^®  ®"  environment  conducive 
recognize  and  deal  with  f^e  TaotltsT'" 
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—All  conditions  were  not  addressed  in  the  contract 
and/or  specifications. 

—Delays  occurred  because  of  untimely  management  deci- 
s  X  On  5 • 

put^d^t^*^^  were  not  made  to  insure  the  quality  of  in- 

1.  to  properly  manage  the  software  contract  re¬ 

sulted  in; 

—The  loss  of  financial  control  of  the  agency  function. 

million  for  systems  analysis 
and  arbitration  support  from  outside  consultants. 

aK?  additional  cost  to  bring  the  system  up  to  accept¬ 
able  levels  of  performance. 

Why  this  contract  failed 

V  factors  jeopardized  the  success  of  this  software 
contract  from  the  start.  First  the  system  specifications 
were  incomplete  when  the  contract  was  let  and  they  were 
never  completed  by  anyone.  Second,  the  contractor's  work 
was  reviewed  poorly  or  not  at  all. 

fi^™  reviewed  the  system  it  concluded  that 
the  first  step  necessary  to  bring  the  system  up  to  par  was 
to  complete  the  specifications  "because  the  system  was  never 

specifications  were  only 

ma  complete.  Also,  the  specifications  implied  too 

many  things  rather  than  being  specific  about  any  of  them. 

The  incomplete  system  specifications  indicated  that  the  aqen- 

the 

Thei”°in^th^®  specifications  from  the  previous  contract, 
hv  contract,  a  system  which  was  approved 

at  the  completion  of  each  developed  stage 
showed  serious  problems  immediately  after  installation. 
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Case  Stud^ 


The  engineering  technical  information 
system  that  was  developed  from  inadequate 
technical  information 


ma-inr^^Mha,  L  ;  Contract  was  for  the  development  of  a 

of  ^  large  automated  system  designed  to  main¬ 
tain  and  retpeve  engineering  data  and  related  information 
to  support  the  agency's  engineering  and  procurement  function 
The  subsystem  is  designed  to  provide  data  to  alS  englneerr 
^®oonstructing  the  original  configuration  of  a  major  part 
or  hardware  assembly  before  modifications  were  introduLr 

subsystem  conttacf to 

be  593,039  and  28  months.  Contract  modifications  and  cost 
overruns  increased  the  final  cost  to  $123,726  in  28  months. 

What  happened  during  the  contract 


with  develop  a  formalized  plan  along 

<3®tailed  procedures  for  implementation  as  well  as  com¬ 
puter  programs  needed  to  establish  the  subsystem.  The  sub- 

Jk  as  a  modiflcatiortr;,.  JarUer 

contract.  The  subsystem  contract  did  not  define  standards  or 

vidl^^i?th°*^  measuring  product  quality.  Timetables  were  pro¬ 
vided  within  the  contract  for  completion  of  individual  re¬ 
quirements.  Although  no  standard  test  data  package  existed 

ftrat^thrabilitrof^th^^®  contractor  was  required  to  demon¬ 
strate  the  ability  of  the  computer  programs  to  operate  and 

to  correct  or  replace  any  unsatisfactory  work. 

printed  three  reports.  Agency  officials 
stated  that  requirements  were  inadequately  defined  for  one 

cost^of^$7°902‘hn^  contract  modification  was  required  at  a 
cost  of  $7,902  to  correct  this  deficiency.  About  18  months 
into  the  contract  period,  a  second  modification  wafnegSti- 
ated  which  cost  $19,044.  of  this  amount,  about  $J?,SoO  was 
for  correcting  a  deficiency  and  the  rest  for  adding  another 
function.  An  agency  programmer  stated  that  before^the  outputs 
could  be  used  he  had  to  write  auxiliary  compute?  program^ 
w  ose  output  supplemented  the  information  shown  on  the 
contractually-required  reports. 

contractor  did  not  meet  specified  comple- 
fo?  ff?l??:  payments  to  ?he  coTtrlttlr 

System  documentation  seemed  to  be  adeauate  hM^  ^h.a 
computer  programs  themselves  were  not  p^ollr^rdoc^e^Jla. 
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Why  this  contract  failed 

The  agency  did  not  perform,  or  contract  for,  adequate 
system  analysis  work.  This  is  indicated  by  the  fact  that 
specifications  were  not  adequately  defined  to  assure  that 
the  software  would  have  the  necessary  capabilities.  Quality 
assurance  and  testing  procedures  in  the  contract  were  inade¬ 
quate  to  assure  that  the  software  would  meet  user  needs 
without  modification  by  agency  programmers.  Documentation 
standards  should  have  made  program  documentation  mandatory. 
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Case  Study  9  -  The  court  case  retrieval  system 

A  Federal  agency  provided  funds  to  allow  a  county  to 
develop  computer  software  to  automate  the  criminal  court 
calendar /  speed  the  scheduling  of  hearings,  and  to  provide 
current  case  and  defendant  information  to  Federal,  State, 
county,  and  local  agencies.  The  county  let  a  fixed-fee  con¬ 
tract  costing  $74,405  for  the  development  of  this  software. 

The  contract  was  to  last  12  months  but  actually  took  about 
36  months.  County  programmers  later  did  about  $16,000  worth 
of  extra  work,  and  the  programs  were  eventually  used. 

What  happened  during  the  contract 

The  county  did  not  do  a  significant  amount  of  development 
work  before  contracting,  so  the  contractor  had  to  establish 
system  requirements— —one  of  the  initial  steps  in  software  de¬ 
velopment.  No  contract  provision  existed  to  allow  the  county 
to  review  the  contractor's  work  in  this  initial  step  and  can¬ 
cel  the  rest  of  the  contract  if  not  satisfied.  The  contract 
required  that  all  specifications  and  documentation  conform 
to  the  county's  data  processing  standards;  however,  county 
programmers  said  that  no  written  program  standards  existed 
when  the  contract  was  signed*  The  county  progranufners  did 
say  that  unwritten  standards  were  discussed  with  the  contrac¬ 
tor,  but  because  the  contractor  did  all  the  coding  off  county 
premises,  the  county  could  not  review  the  code  before  deliverv 
to  ensure  compliance.  ^ 

r-'s.. 

The  contract  included  timetables  for  completion  of  in¬ 
dividual  requirements;  however,  work  under  the  contract  and 
subsequent  closing  agreement  was  about  3  years  later  than 
originally  negotiated.  The  original  contract  provided  that 
the  county  would  approve  a  test  and  approve  the  adequacy  of 
program  documentation  before  making  final  payment. 

The  closing  agreement  attached  a  time  limit  for  accept¬ 
ance  by  the  county.  It  required  that  the  contractor  programs 
be  tested  and  evaluated  no  later  than  2  months  after  submis¬ 
sion.  Also,  the  closing  agreement  required  that  if  items 
were  found  to  be  unacceptable,  the  contractor  would  make  any 
corrections  or  modifications  quickly,  if  any  items  were  not 
corrected  by  the  contractor  2  months  after  the  date  of  first 
receipt  of  all  items,  the  county  was  to  pay  only  that  propor¬ 
tion  of  the  remaining  moneys  due  for  those  items  that  were 
accepted.  Full  payment  was  not  to  be  made  until  all  items 
were  tested  and  accepted. 

The  contract  contained  a  "termination  or  curtailment" 
clause  which  stated  that  if  the  contractor  failed  to  perform 
any  of  the  requirements,  the  county  could  terminate  the 
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Also,  the  contract  required  the  contractor  to: 

— Sutenit  a  report  analyzing  requirements  and  general 
reconunendations  to  improve  or  revise  the  existing 
criminal  court  procedures. 

— Develop  a  detailed  system  to  satisfy  the  above  report. 

Generate  specifications  and  documentation  for  use  by 
the  agency's  data  processing  office. 

The  report  was  to  include  existing  procedures  for  han¬ 
dling  the  flow  of  information  through  the  court.  Also,  flow¬ 
charts  of  existing  methods  of  processing  cases  and  defendants 
and  an  analysis  of  forms  and  documents  used  by  the  county  were 
required.  The  contractor  was  also  to  recommend  a  new  system 
concept,  including  descriptions,  flowcharts,  new  and  revised 
forms,  and  a  general  approach  to  a  data-processing-oriented 
system  of  information  retrieval  for  the  county. 

The  contract  listed  numerous  items  which  were  to  be  in¬ 
cluded  in  or  delivered  with  the  systems  specifications,  the 
programming  requirements,  and  the  documentation.  The  systems 
specification  items  included  a  system  requirements  document 
which  was  to  specify  production  and  user  delivery  dates,  work¬ 
load,  and  controls;  description  of  major  computer  and  clerical 
processes;  and  narrative  showing  the  flow  of  documents,  hand 
and  machine  processing,  and  documentation  of  ident^i,fied  user 
needs.  The  programming  requirements  included  adherence  to 
county  standards  for  programming  language  (a  choice  of  two 
was  allowed),  and  the  delivery  of  decks  and  listings.  The 
documentation  requirements  list  included  functional  flowcharts, 
program  flowcharts,  narrative  descriptions,  file  layouts,  and 
a  data  element  dictionary. 

However,  despite  the  numerous  items  named  in  the  lists, 
the  contract  contained  nothing  specific  describing  how  the 
items  were  to  look,  and  the  county  standards  cited  did  not 
exist  on  paper. 

County  officials  stated  that  the  contractor's  first  sys¬ 
tem  proposal  was  disapproved  because  it  appeared  that  the 
contractor  merely  mechanized  the  existing  manual  system 
instead  of  streamlining  procedures  to  exploit  the  computer. 

The  contractor  submitted  a  revised  system  proposal  which 
the  county  approved. 

One  county  official  stated  that  the  contractor  claimed 
criminal  court  system  expertise  that  he  did  not  have. 

Another  official  cited  examples  of  the  contractor's  poor 
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understanding  of  court  procedures.  Contractor  representatives 
perforraln?!?"  demonstrated  a  "less  than  perfect 

I  contract,  the  par- 

^  closing  agreement."  The  contractor 
agreed  to  complete  work  on  specified  contract  items  while 
the  county  agreed  to  complete  all  other  contract  requirements 
plus  those  additional  items  not  found  in  the  initial  eont-yani- 

monthly  and  annual  reports.  A  county  official  estimated  tLt 
It  would  cost  the  county  an  additional  $15,000  to  $18,000  to 
write  the  programs  themselves.  9  r  uo  to 

tion  f  originally  negotiated  coraple- 

tion  date,  the  contractor  completed  and  the  county  accepted 
all  Items  under  the  terms  of  the  closing  agreement.  ^ 

County  officials  said  that  most  of  the  contractor's  com- 

puter  programs — about  60  percent  of  the  program  statement _ 

rewritten  by  the  county.  Prog?Ls  alsl  ?l|S“ed 
extensive  error  correction  1/  before  they  would  work  properlv 
The  software  documentation  did  not  include  program  loqic^floJ- 
charts  or  sufficient  explanation  of  complex  processing  logicT 

Why  this  contract  failed 

which^laterfovLffh^  lacked  specificity.  Documentation, 
wnicn  later  proved  to  be  a  problem,  was  required  to  make  the 

system  conform  to  data  processing  standards  of  the  county 

trwt^wariet^  ^Al,o°  written  program  standards  when  the^ron- 
tpct  was  let.  Also,  since  the  contractor's  first  DroDo«?ei 
showed  no  understanding  of  the  county's  mission,  the  counf 
apparently  did  not  furnish  the  contractor  with  adequate  fuL- 
tional  specifications.  Another  indication  of  the  lack  of 

that  the  contractor  was  allowed  to  use  either 
of  two  progranuning  languages* 

county  also  had  minimal  monitoring,  review,  and 
quality  assurance  procedures  in  effect.  This  wlfuiustrated 
softSLf  dates,  the  fact  that  much  of  the 

nnS  the  in-house  cost  of  from 

$18,000  to  do  work  which  was  originally  the  con¬ 
tractor's  responsibility.  These  are  all  effects  of  o^or 
contractor  performance  and  poor  county  contract  management. 


1/  called  debugging 
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General 

Services 

Administration  Washington.  DC  20405 


JOL  I  i 


Honorable  Elmer  Staats 
Comptroller  General  of  the  United  States 
General  Accounting  Office 
Washington,  D.C.  20548 

Dear  Mr.  Staats: 

We  have  reviewed  the  General  Accounting  Office  draft  report  entitled 
"Contracting  for  Computer  Software  Development  --  Serious  Problems 
Require  Management  Attention  to  Avoid  Wasting  Additional  Millions,” 
dated  May  24,  1979,  and  agree  with  its  conclusion  and  recommendations. 

The  following  comments  are  offered  in  response  to  specific  recommen¬ 
dations: 

1.  ”...  that  NBS  and  GSA  collaborate  to  issue  specific  guidance 
to  assist  Federal  agencies  .  .  .”  (page  28).  As  part  of  its 
Contract  Services  Program  our  Automated  Data  and  Tele¬ 
communications  Service  (ADTS)  is  developing  contracting  guide¬ 
lines  for  its  own  use  in  providing  software  development  services 
to  other  Federal  agencies.  We  welcome  the  opportunity  to 
collaborate  with  NBS  on  extending  the  benefits  of  these  guide¬ 
lines  to  other  agencies. 

2.  ”0ne  method  to  provide  more  specific  guidance  to  the  working 
level  project  manager  would  be  for  NBS  and  GSA  together  to 
design  a  series  of  model  contracts  .  .  (page  29).  ADTS  is 
also  developing  Standard  Terms  and  Conditions  for  software 
development  contracting  as  part  of  its  Contract  Services 
Program,  and  would  like  to  collaborate  with  NBS  in  this  effort. 

3.  .  .  that  Federal  agencies  which  contract  extensively  for 
software  development  train  project  managers  in  the  pertinent 
software,  contract  and  management  skills  needed”  (page  29).  In 
its  Manpower  Services  Program,  ADTS  provides  software  develop¬ 
ment  project  managers  to  other  Federal  agencies  on  a  reimburs¬ 
able  basis.  As  in  the  two  areas  mentioned  above,  we  welcome  the 
chance  to  share  our  experience  with  other  agencies,  by  providing 
on-the-job  or  formal  training,  or  both. 
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Our  Automated  Data  Telecommunications  Service  (ADTS)  will  be  available 
to  work  with  the  National  Bureau  of  Standards  to  implement  the  report's 
recommendations.  t 


Sincerely 


K,  G»  Freeman 

Administrator 


GAO  note;  The  page  numbers  have  been  changed  to  correspond 
to  the  pages  in  the  final  report. 
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UNITED  STATES  DEPARTMENT  DF  CDMMERCE 
Office  of  Inspector  Generel 

Washington.  D  C.  20230 


July  18p  1979 


Mr.  Henry  Eschwege 
Director,  Community  and  Economic 
Development  Division 
U.  S.  General  Accounting  Office 
Washington,  D.  C.  20548 

Dear  Mr.  Eschwege: 

This  is  in  reply  to  your  letter  of  May  31,  1979  requesting  Comments 
on  the  draft  report  entitled  "Contracting  for  Computer  Software 
Development  —  Serious  Problems  Require  Management  Attention  to 
Avoid  Wasting  Additional  Millions".  The  Assistant  Secretary  for 
Science  and  Technology  provided  the  enclosed  comments. 

In  addition,  the  Office  of  Procurement  and  ADP  Management  concurs 
with  the  recommendations  that  Federal  agencies  Involved  in  software 
contracting  (1)  train  project  managers  in  pertinent  skills  necessary 
to  manage  large  software  development  contracts  and  (2)  ensure  that 
appropriate  action  is  taken  In  each  phase  of  contracting  for  soft¬ 
ware  development.  They  have  identified  several  ways  In  which  the 
Office  of  Procurement  and  ADP  Management,  as  the  Department's 
central  procurement  activity,  can  help  ensure  proper  and  effective 
contracting  for  software  development.  First,  they  are  planning  a 
series  of  procurement  briefings  to  be  conducted  at  each  Department 
operating  unit  to  better  educate  project  managers  on  ADP  procurement 
regulations  and  techniques.  Secondly,  they  are  disseminating  through¬ 
out  the  Department  the  provisional  checklist  for  software  development 
contracting  which  was  provided  as  Appendix  I  to  the  GAO  report.  Third, 
they  recently  published  and  distributed  guidelines  for  the  preparation 
of  ADP  solicitation  technical  packages. 

We  have  reviewed  the  comments  and  believe  they  are  responsive  to  the 
matters  discussed  in  the  report. 


Enclosure 
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UNITED  STATES  DEPARTMENT  OF  COMMERCE 
Th«  Assistant  Sacratary  for  Scienea  and  Technology 

Washington,  D  C  20230  (202)  377-31 1 1 


4 


JUL  3  1379 


Mr.  Henry  Eschwege 
Director,  Community  and  Economic 
Development  Division 
U.S.  General  Accounting  Office 
Washington,  D.C.  205A8 

Dear  Mr*  Eschwege: 

Thank  you  for  the  opportunity  to  review  the  draft  GAO  report  entitled 
"Contracting  For  Computer  Software  Development  —  Serious  Problems  Require 
Management  Attention  To  Avoid  Wasting  Additional  Millions."  We  generally 
agree  with  the  scope  of  the  problem  outlined  in  the  draft  report,  and  share 
GA0*s  concern. 

Many  of  the  program  products  in  the  Congressionally-approved  Federal  ADP 
standards  plan  for  NBS  are  applicable  to  problems  identified  by  this  report, 
especially  those  problems  characteristic  of  the  software  development  pro¬ 
cess  itself.  Products  such  as  documentation  guidelines,  language  standards, 
cost  estimation  guidelines,  programming  guidelines,  and  testing  guidelines 
can  all  be  applied  to  software  developed  under  contract  for  the  Federal 
Government,  as  well  as  to  in-house  development  and,  in  many  cases,  procure¬ 
ment  of  software  products.  Additional  guidelines  applicable  to  the  unique 
problems  of  contracting  for  custom  software  are  not  currently  in  the 
Congressionally-approved  program  plan. 

The  NBS  staff  has  prepared  the  following  specific  comments  which  we  hope 
will  be  of  use  to  you  as  the  final  report  is  prepared. 

1.  Page  5,  middle:  In  the  first  sentence  under  Software  development, 
add  "requirements  analysis,  system  design  or  specification,  and 
after  the  word  "into."  The  corrected  sentence  should  read:  "The 
life  cycle  of  computer  software  is  divided  into  requirements 
analysis,  system  design  or  specification,  and  development  and 
operation." 

2.  Page  6,  middle: The  sentence  at  the  .top  of  page  six  and  the  sentence 

at  the  middle  of  page  six  are  incomplete.  It  appears  that  some  material 
is  missing  from  the  text. 
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Page  15,  top:  The  responsibilities  cited  in  the  first  paragraph 
are  inconsistent  with  the  responsibilities  assigned  to  NBS  under 
the  Brooks  Bill  (PL  89-306)  in  Section  111  (f).  We  reconanend 
that:  (1)  the  words  "and  guidelines"  be  inserted  after  the  word 

"standards"  in  the  fourth  line  from  the  top  of  the  page,  and  (2) 
the  remainder  of  the  sentence  and  the  paragraph  be  deleted.  The 
revised  sentence  would  read:  "The  National  Bureau  of  Standards 
(NBS)  is  assigned  the  task  of  developing  technical  standards 
and  guidelines,** 

Page  15,  bottom:  Delete  the  sentence  **NBS  representatives 
recently  informed  us  that  their  proposed  new  budget  Includes 
a  project  on  Software  development  contracting." 

Page  37,  middle:  Add  the  following  statement  as  the  third  item 
under  PROCEDURES.  •  "The  contract  should  provide  for  adequate  ^  ^ 

phase  over  of  software  from  the  contractor  to  the  agency,  includ¬ 
ing  such  items  as  agency  training  and  walk-throughs." 

Page  39,  middle:  Delete  the  second  item  under  METHODS  and  insert 
the  following  statement.  "The  contract  should  also  provide 
separate  due  dates  and  costs  for  each  deliverable  and  a  provision 
to  withhold  payment  for  incomplete  or  unacceptable  work. 

Page  41.  bottom:  Add  the  following  statements  to  the  fourth  item 
under  POSSIBLE  WAYS  OF  ACCOMPLISHING  THE  PROCEDURES.  However, 
undocumented  informal  communication  can  lead  to  cost  overruns. 

Any  changes  in  the  scope  of  work  must  be  handled  by  formal  modifi¬ 
cation  of  the  contract  document." 

General  Comment:  Appendix  I  does  not  adequately  provide  for  the 
process  of  evaluating  proposals  of  software  contractors  as  a 
separate  phase.  Suggested  items  to  be  used  in  evaluations 
should  be  developed  extensively  and  should  include  the  following 
elements: 

o  Is  the  expertise  of  the  contractor  commensurate  with 
the  complexity  of  the  problem? 

o  Does  the  contractor  have  experience  with  the  software 
and  hardware  to  be  used  during  development  or  will 
training  (possibly  at  Government  expense)  be  required? 
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o  What  effect  on  contract  start-up  time  does  the  contractor's 
experience  have? 

This  separate  phase  should  be  added  after  page  39  which  covers 
CONTRACTING  and  before  page  41  which  covers  PERFORMANCE  PERIOD, 


We  believe  that  this  report  will  prove  valuable  to  the  Federal  community 
in  improving  its  contracting  practices  with  respect  to  computer  software 
development . 

n 

Sincerely', 

_ 

/  Jordan  J.  Baruch 


GAO  note:  The  page  numbers  have  been  changed  to  correspond 
to  the  pages  in  the  final  report. 
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